Centralized network control for a cloud-based services exchange

ABSTRACT

In some examples, a method includes: providing, by a centralized network control (CNC) system, a software interface to receive service requests for configuration of services within an edge network of one or more network data centers that are controlled by the CNC system; receiving a service request to configure a service within the edge network; generating a service definition that specifies service requirements to implement the service; determining at least one network field unit that is capable of servicing the service request; and sending, to the at least one network field unit, the service definition to configure the service at the portion of the edge network.

This application claims the benefit of U.S. Provisional Application No.62/160,547, filed May 12, 2015, the entire content of which isincorporated by reference herein.

TECHNICAL FIELD

The invention relates to computer networks and, more specifically, tofacilitating service provisioning and delivery among cloud servicecustomers and cloud service providers.

BACKGROUND

Cloud computing refers to the use of dynamically scalable computingresources accessible via a network, such as the Internet. The computingresources, often referred to as a “cloud,” provide one or more servicesto users. These services may be categorized according to service types,which may include for examples, applications/software, platforms,infrastructure, virtualization, and servers and data storage. The namesof service types are often prepended to the phrase “as-a-Service” suchthat the delivery of applications/software and infrastructure, asexamples, may be referred to as Software-as-a-Service (SaaS),Platform-as-a-Service (PaaS), and Infrastructure-as-a-Service (IaaS),respectively.

The term “cloud-based services” or, more simply, “cloud services” refersnot only to services provided by a cloud, but also to a form of serviceprovisioning in which cloud customers contract with cloud serviceproviders for the online delivery of services provided by the cloud.Cloud service providers manage a public, private, or hybrid cloud tofacilitate the online delivery of cloud services to one or more cloudcustomers.

SUMMARY

In general, this disclosure describes a programmable network platformfor dynamically programming a cloud-based service exchange (“cloudexchange”) to responsively and assuredly fulfill service requests thatencapsulate business requirements for services provided by the cloudexchange and/or cloud service providers coupled to the cloud exchange.The programmable network platform as described herein may, as a result,orchestrate a business-level service across heterogeneous cloud serviceproviders according to well-defined service policies, quality ofservice, service level agreements, and costs, and further according to aservice topology for the business-level service.

The programmable network platform enables the cloud service providerthat administers the cloud exchange to dynamically configure and managethe cloud exchange to, for instance, facilitate virtual connections forcloud services delivery from multiple cloud service providers to one ormore cloud customers. The cloud exchange may enable cloud customers tobypass the public Internet to directly connect to cloud servicesproviders so as to improve performance, reduce costs, increase thesecurity and privacy of the connections, and leverage cloud computingfor additional applications. In this way, enterprises, network carriers,and SaaS customers, for instance, can at least in some aspects integratecloud services with their internal applications as if such services arepart of or otherwise directly coupled to their own data center network.

In some aspects, a programmable network platform as described hereinoperates according to a distributed model in which a centralized networkcontroller (CNC) manages globally-distributed and intelligent logic inthe form of network field units (NFUs). The CNC may receive a businessservice request via an interface and convert the business servicerequest into business instantiation parameters and network provisioningparameters to be delivered and assured as a business service within thecloud exchange. The CNC thus operates as a central intelligentprocessing unit of the programmable network platform. Each instantiationof a programmable network platform may have one logical instance of thisintelligent logic (i.e., the CNC). The CNC may provide service assuranceusing a Monitor, Analyze, Plan and Execute (MAPE) loop methodology andis implemented to ensure the service level agreements are adhered to bythe service.

The various NFUs are distributed among globally-distributed cloudexchange points of a cloud exchange provider that administers theprogrammable network platform. Each NFU receives network instantiationcommands/parameters from the CNC and instantiates and configures thenetwork resource(s) that is needed to deliver the service. The NFU hasthe intelligence to deliver and assure network services according to CNCrequests. In some aspects, the NFU further has the capability ofcommunicating with a third party orchestration system, if needed by theservice request. The NFU applies a separate MAPE loop to ensure that thenetwork services delivered by the unit is assured for the life cycle ofthe service

In some aspects, a programmable network platform described herein mayprovide for orchestrating a service that involves both native andthird-party components as single service while ensuring policy,security, and SLA consistency. The programmable network platform mayorchestrate the third-party service components using a third-party (or“partner”) orchestration module (or “plugin”). A third-partyorchestration module allows a third-party orchestration system toregister its capabilities (e.g., service catalog, policy, security andSLA) with the programmable network platform. The cloud service provider,as the service owner, may use the programmable network platform todirect the third-party orchestration system, via the correspondingthird-party orchestration module, as part of the workflow for theservice delivery to stand-up and deliver a third-party service for theservice.

In some aspects, the programmable network platform described herein, mayprovision a cloud exchange to deliver services made up of multipleconstituent services provided by multiple different cloud serviceproviders. Each of these constituent services is referred to herein as a“micro-service” in that it is part of an overall service applied toservice traffic. That is, a plurality of micro-services may be appliedto service traffic in a particular “arrangement,” “ordering,” or“topology,” in order to make up an overall service for the servicetraffic. The micro-services themselves may be applied or offered by thecloud service providers.

The programmable network platform may in this way orchestrate abusiness-level service across heterogeneous cloud service providers. Theprogrammable network platform exposes interfaces by which a portal,console (e.g., user interface application), or other application maydefine the service policy, quality, SLAs, and cost as a coordinatedservice topology made up of micro-services provided by different cloudservice providers (or “cloud vendors”). Each micro-service may have acorresponding service policy, quality, SLA, and cost, as part of theoverall, end-to-end business service definition. When provided with aservice definition for an end-to-end service having multiple componentmicro-services, the programmable network platform orchestrates each ofthe micro-services within the cloud exchange and stitches themicro-services together according to the defined topology in order toreify the end-to-end service within the cloud exchange data plane (e.g.,an edge network for the cloud exchange). As a result, the cloud exchangeinterconnects, in the data plane, micro-services provided by respectivecloud services providers on behalf of and for the benefit of a customerof the cloud exchange. In doing so, the cloud exchange provider mayfacilitate business transactions between the cloud service providers andcustomers.

In some aspects, when provided with a service definition for anend-to-end service having multiple component micro-services, aprogrammable network platform as described herein may orchestrate eachof the micro-services within the cloud exchange and stitch themicro-services together according to the defined topology in order toreify the end-to-end service within the cloud exchange. In accordancewith techniques of this disclosure, the service definition for anend-to-end service may enable a user of the programmable networkplatform to define not only the end-to-end service but also the servicetopology in such a ways as to ensure the correct sequencing of themicro-services service chain. The data encapsulated in the data modelfor the service definition may also include the authoritative serviceowner for business purposes (e.g., billing and SLA assurance). The“user” may refer to a customer, the cloud exchange provider, or a cloudservice provider that is the authoritative service owner.

By using a data model for a multi-cloud, multi-service servicedefinition as described herein, the programmable network platform(and/or other orchestration systems such as software-defined networking(SDN) controllers or orchestrators) may be enabled to recognize aservice request as a request for a set of micro-services that make upthe entire service. In some examples, the service definition includesseveral sections that will enable the programmable network platform toprovide the service of chaining several services, whether of nativeservices provided by the cloud exchange provider or of cloud servicesprovided one or multiple cloud service providers. That is, the cloudexchange provider that administers the programmable network platform isable to provide a chaining service that, when given respectivedefinitions for multiple micro-services and a topology (or sequence) forthe multiple micro-services, interconnects the micro-services accordingto the topology to facilitate an end-to-end service. The data model thusprovides data with which the programmable network platform caneffectively instantiate the requested chain of services and to alsoensure that the services thus rendered are chained in the correcttopology. The data model may be divided by the programmable networkplatform into one or more service requests that the native programmablenetwork platform for the cloud exchange may issue to other serviceorchestration systems to complete. Other service orchestration systemsmay include, e.g., SDN controllers and/or orchestration systems forcloud service providers that facilitate NFV-instantiation and servicetraffic routing to/from NFV instances.

In some examples, a method includes: providing, by a centralized networkcontrol (CNC) system, a software interface to receive service requestsfor configuration of services within an edge network of one or morenetwork data centers that are controlled by the CNC system; receiving,by the CNC system and via the software interface, a service request toconfigure a service within the edge network of the network data center,wherein the edge network within the one or more network data centersconnects through one or more switching fabrics of the one or morenetwork data centers; generating, by the CNC system and based on theservice request, a network service definition that specifies one or moreservice requirements to implement the service; determining, by the CNCsystem and based on the network service definition, at least one networkfield unit that is capable of servicing the service request, wherein thenetwork field unit controls a portion of the edge network, wherein thenetwork service definition is usable by the at least one network fieldunit to configure the portion of the edge network to provide theservice; and sending, by the CNC system and to the at least one networkfield unit, the network service definition to configure at least onenetwork service of the service at the portion of the edge network.

In some examples, a centralized network control (CNC) system includes:one or more computer processors; and a memory comprising instructionsthat when executed by the one or more computer processors cause the oneor more computer processors to: provide a software interface to receiveservice requests for configuration of services within an edge network ofone or more network data centers that are controlled by the CNC system;receive, via the software interface, a service request to configure aservice within the edge network of the network data center, wherein theedge network within the one or more network data centers connectsthrough one or more switching fabrics of the one or more network datacenters; generate, based on the service request, a network servicedefinition that specifies one or more service requirements to implementthe service; determine, based on the network service definition, atleast one network field unit that is capable of servicing the servicerequest, wherein the network field unit controls a portion of the edgenetwork, wherein the network service definition is usable by the atleast one network field unit to configure the portion of the edgenetwork to provide the service; and send, to the at least one networkfield unit, the network service definition to configure at least onenetwork service of the service at the portion of the edge network.

In some examples, a non-transitory computer readable medium comprisesinstructions that, when executed, cause at least one programmableprocessor of a centralized network control (CNC) system to performoperations including: providing a software interface to receive servicerequests for configuration of services within an edge network of one ormore network data centers that are controlled by the CNC system;receiving, via the software interface, a service request to configure aservice within the edge network of the network data center, wherein theedge network within the one or more network data centers connectsthrough one or more switching fabrics of the one or more network datacenters; generating, based on the service request, a network servicedefinition that specifies one or more service requirements to implementthe service; determining, based on the network service definition, atleast one network field unit that is capable of servicing the servicerequest, wherein the network field unit controls a portion of the edgenetwork, wherein the network service definition is usable by the atleast one network field unit to configure the portion of the edgenetwork to provide the service; and sending, to the at least one networkfield unit, the network service definition to configure at least onenetwork service of the service at the portion of the edge network.

The details of one or more embodiments of the invention are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages of the invention will be apparent from thedescription and drawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram that illustrates a high-level view of a datacenter that provides an operating environment for a cloud-based servicesexchange.

FIG. 2 is a block diagram illustrating a high-level view of a datacenter that provides an operating environment for a cloud-based servicesexchange, according to techniques described herein.

FIGS. 3A-3B are block diagrams illustrating example networkinfrastructure and service provisioning by a programmable networkplatform for a cloud exchange that aggregates the cloud services ofmultiple cloud service providers for provisioning to customers of thecloud exchange provider and aggregates access for multiple customers toone or more cloud service providers, in accordance with techniquesdescribed in this disclosure.

FIG. 4 is a block diagram illustrating an example of a data center-basedcloud exchange point in which routers of the cloud exchange point areconfigured by programmable network platform with VPN routing andforwarding instances for routing and forwarding aggregated servicetraffic from multiple cloud service provider networks to a customernetwork, according to techniques described herein.

FIG. 5 is a block diagram illustrating a platform for a softwarecontrolled network, the platform operating in accordance with one ormore techniques of the present disclosure.

FIG. 6 is a block diagram illustrating an example service provisioningengine, in accordance with one or more techniques of the presentdisclosure.

FIG. 7 is a block diagram illustrating an example service assuranceengine, in accordance with one or more techniques of the presentdisclosure.

FIG. 8 is a block diagram illustrating an example network provisioningengine, in accordance with one or more techniques of the presentdisclosure.

FIG. 9 is a block diagram illustrating an example network assuranceengine, in accordance with one or more techniques of the presentdisclosure.

FIG. 10 is a block diagram illustrating a programmable network platform,in accordance with one or more techniques of the present disclosure.

FIG. 11 is a block diagram illustrating an example user interface torequest a service, in accordance with one or more techniques of thepresent disclosure.

FIG. 12 is a block diagram illustrating an example user interface todisplay a cost estimate for a service, in accordance with one or moretechniques of the present disclosure.

FIG. 13 is a conceptual diagram illustrating example components for aprogrammable network platform operating according to techniquesdescribed in this disclosure.

FIG. 14A is a block diagram that illustrates an example configuration ofa programmable edge network that has been configured to apply multiplenative services to cloud service traffic aggregated by a cloud exchangefrom multiple cloud service providers for delivery to a customer.

FIG. 14B is a block diagram that illustrates an example configuration ofa programmable edge network that has been configured to offer anend-to-end service that is a sequence of multiple constituentmicro-services applied by respective cloud service providers.

FIG. 15 is a conceptual diagram illustrating interfaces among componentsfor programming a cloud exchange using a programmable network platformaccording to techniques described in this disclosure.

FIG. 16 is a block diagram illustrating a programmable network platformthat includes interfaces by which external applications may configure acloud exchange to facilitate delivery of cloud services from cloudservice providers according to techniques described in this disclosure.

FIG. 17 is a block diagram illustrating further details of one exampleof a computing device that operates in accordance with one or moretechniques of the present disclosure.

Like reference characters denote like elements throughout the figuresand text.

DETAILED DESCRIPTION

In general, this disclosure describes a programmable network platformfor real-time configuration and management of a cloud-based servicesexchange (“cloud exchange”). As described herein, the interconnectionplatform provides customers of the exchange, e.g., enterprises, networkcarriers, and SaaS customers, with secure, private, virtual connectionsto multiple cloud service providers (CSPs) globally. The multiple CSPsparticipate in the cloud exchange by virtue of their having at least oneaccessible port in the cloud exchange by which a customer can connect tothe one or more cloud services offered by the CSPs, respectively.

According to various examples described herein, a cloud exchange isdescribed that allows private networks of any customer to be directlycross-connected to any other customer at a common point, therebyallowing direct exchange of network traffic between the networks of thecustomers. Customers may include network carriers (or network serviceproviders), enterprises, and other users of cloud services offered byone or more cloud service providers.

FIG. 1 illustrates a conceptual view of a network system 2 having ametro-based cloud exchange that provides multiple cloud exchange pointsaccording to techniques described herein. Each of cloud-based servicesexchange points 128A-128D (described hereinafter as “cloud exchangepoints” and collectively referred to as “cloud exchange points 128”) ofcloud-based services exchange 100 (“cloud exchange 100”) may represent adifferent data center geographically located within the samemetropolitan area (“metro-based,” e.g., in New York City, N.Y.; SiliconValley, Calif.; Seattle-Tacoma, Wash.; Minneapolis-St. Paul, Minn.;London, UK; etc.) to provide resilient and independent cloud-basedservices exchange by which cloud-based services customers (“cloudcustomers”) and cloud-based service providers (“cloud providers”)connect to receive and provide, respectively, cloud services. In variousexamples, cloud exchange 100 may include more or fewer cloud exchangepoints 128. In some instances, a cloud exchange 100 includes just onecloud exchange point 128. As used herein, reference to a “cloudexchange” or “cloud-based services exchange” may refer to a cloudexchange point. A cloud exchange provider may deploy instances of cloudexchanges 100 in multiple different metropolitan areas, each instance ofcloud exchange 100 having one or more cloud exchange points 128.

Each of cloud exchange points 128 includes network infrastructure and anoperating environment by which cloud customers 108A-108D (collectively,“cloud customers 108”) receive cloud services from multiple cloudservice providers 110A-110N (collectively, “cloud service providers110”). Cloud customers 108 may receive cloud-based services directly viaa layer 3 peering and physical connection to one of cloud exchangepoints 128 or indirectly via one of network service providers 106A-106B(collectively, “NSPs 106,” or alternatively, “carriers 106”). NSPs 106provide “cloud transit” by maintaining a physical presence within one ormore of cloud exchange points 128 and aggregating layer 3 access fromone or customers 108. NSPs 106 may peer, at layer 3, directly with oneor more cloud exchange points 128 and in so doing offer indirect layer 3connectivity and peering to one or more customers 108 by which customers108 may obtain cloud services from the cloud exchange 100. Each of cloudexchange points 128, in the example of FIG. 1, is assigned a differentautonomous system number (ASN). For example, cloud exchange point 128Ais assigned ASN 1, cloud exchange point 128B is assigned ASN 2, and soforth. Each cloud exchange point 128 is thus a next hop in a path vectorrouting protocol (e.g., BGP) path from cloud service providers 110 tocustomers 108. As a result, each cloud exchange point 128 may, despitenot being a transit network having one or more wide area network linksand concomitant Internet access and transit policies, peer with multipledifferent autonomous systems via external BGP (eBGP) or other exteriorgateway routing protocol in order to exchange, aggregate, and routeservice traffic from one or more cloud service providers 110 tocustomers. In other words, cloud exchange points 128 may internalize theeBGP peering relationships that cloud service providers 110 andcustomers 108 would maintain on a pair-wise basis. Instead, a customer108 may configure a single eBGP peering relationship with a cloudexchange point 128 and receive, via the cloud exchange, multiple cloudservices from one or more cloud service providers 110. While describedherein primarily with respect to eBGP or other layer 3 routing protocolpeering between cloud exchange points and customer, NSP, or cloudservice provider networks, the cloud exchange points may learn routesfrom these networks in other way, such as by static configuration, orvia Routing Information Protocol (RIP), Open Shortest Path First (OSPF),Intermediate System-to-Intermediate System (IS-IS), or other routedistribution protocol.

As examples of the above, customer 108C is illustrated as havingcontracted with a cloud exchange provider for cloud exchange 100 todirectly access layer 3 cloud services via cloud exchange points 128C.In this way, customer 108D receives redundant layer 3 connectivity tocloud service provider 110A, for instance. Customer 108C, in contrast,is illustrated as having contracted with the cloud exchange provider forcloud exchange 100 to directly access layer 3 cloud services via cloudexchange point 128C and also to have contracted with NSP 106B to accesslayer 3 cloud services via a transit network of the NSP 106B. Customer108B is illustrated as having contracted with multiple NSPs 106A, 106Bto have redundant cloud access to cloud exchange points 128A, 128B viarespective transit networks of the NSPs 106A, 106B. The contractsdescribed above are instantiated in network infrastructure of the cloudexchange points 128 by L3 peering configurations within switchingdevices of NSPs 106 and cloud exchange points 128 and L3 connections,e.g., layer 3 virtual circuits, established within cloud exchange points128 to interconnect cloud service provider 110 networks to NSPs 106networks and customer 108 networks, all having at least one portoffering connectivity within one or more of the cloud exchange points128.

In some examples, cloud exchange 100 allows a corresponding one ofcustomer customers 108A, 108B of any network service providers (NSPs) or“carriers” 106A-106B (collectively, “carriers 106”) or other cloudcustomers including customers 108C to be directly cross-connected, via avirtual layer 2 (L2) or layer 3 (L3) connection to any other customernetwork and/or to any of CSPs 110, thereby allowing direct exchange ofnetwork traffic among the customer networks and CSPs 110.

Carriers 106 may each represent a network service provider that isassociated with a transit network by which network subscribers of thecarrier 106 may access cloud services offered by CSPs 110 via the cloudexchange 100. In general, customers of CSPs 110 may include networkcarriers, large enterprises, managed service providers (MSPs), as wellas Software-as-a-Service (SaaS), Platform-aaS (PaaS), Infrastructure-aaS(IaaS), Virtualization-aaS (VaaS), and data Storage-aaS (dSaaS)customers for such cloud-based services as are offered by the CSPs 110via the cloud exchange 100.

In this way, cloud exchange 100 streamlines and simplifies the processof partnering CSPs 110 and customers (via carriers 106 or directly) in atransparent and neutral manner. One example application of cloudexchange 100 is a co-location and interconnection data center in whichCSPs 110 and carriers 106 and/or customers 108 may already have networkpresence, such as by having one or more accessible ports available forinterconnection within the data center, which may represent any of cloudexchange points 128. This allows the participating carriers, customers,and CSPs to have a wide range of interconnectivity options within thesame facility. A carrier/customer may in this way have options to createmany-to-many interconnections with only a one-time hook up to one ormore cloud exchange points 128. In other words, instead of having toestablish separate connections across transit networks to accessdifferent cloud service providers or different cloud services of one ormore cloud service providers, cloud exchange 100 allows customers tointerconnect to multiple CSPs and cloud services.

In accordance with techniques described herein, cloud exchange 100includes a programmable network platform 120 for dynamically programmingcloud exchange 100 to responsively and assuredly fulfill servicerequests that encapsulate business requirements for services provided bycloud exchange 100 and/or cloud service providers 110 coupled to thecloud exchange 100. The programmable network platform 120 as describedherein may, as a result, orchestrate a business-level service acrossheterogeneous cloud service providers 110 according to well-definedservice policies, quality of service policies, service level agreements,and costs, and further according to a service topology for thebusiness-level service.

The programmable network platform 120 enables the cloud service providerthat administers the cloud exchange 100 to dynamically configure andmanage the cloud exchange 100 to, for instance, facilitate virtualconnections for cloud-based services delivery from multiple cloudservice providers 110 to one or more cloud customers 108. The cloudexchange 100 may enable cloud customers 108 to bypass the publicInternet to directly connect to cloud services providers 110 so as toimprove performance, reduce costs, increase the security and privacy ofthe connections, and leverage cloud computing for additionalapplications. In this way, enterprises, network carriers, and SaaScustomers, for instance, can at least in some aspects integrate cloudservices with their internal applications as if such services are partof or otherwise directly coupled to their own data center network.

Programmable network platform 120 may represent an application executingwithin one or more data centers of the cloud exchange 100 oralternatively, off-site at a back office or branch of the cloud provider(for instance). Programmable network platform 120 may be distributed inwhole or in part among the data centers, each data center associatedwith a different cloud exchange point 128 to make up the cloud exchange100. Although shown as administering a single cloud exchange 100,programmable network platform 120 may control service provisioning formultiple different cloud exchanges. Alternatively or additionally,multiple separate instances of the programmable network platform 120 maycontrol service provisioning for respective multiple different cloudexchanges.

In the illustrated example, programmable network platform 120 includes aservice interface (or “service API”) 114 that defines the methods,fields, and/or other software primitives by which applications mayinvoke the programmable network platform 120. The service interface 114may allow carriers 106, customers 108, cloud service providers 110,and/or the cloud exchange provider programmable access to capabilitiesand assets of the cloud exchange 100.

For example and as further described herein, the service interface 114may facilitate machine-to-machine communication to enable dynamicprovisioning of virtual circuits in the cloud exchange forinterconnecting customer and cloud service provider networks. In thisway, the programmable network platform 120 enables the automation ofaspects of cloud services provisioning. For example, the serviceinterface 114 may provide an automated and seamless way for customers toestablish, de-install and manage interconnection with multiple,different cloud providers participating in the cloud exchange.

Further example details of a cloud-based services exchange can be foundin U.S. Provisional Patent Application 62/149,374, filed Apr. 17, 2015and entitled “Cloud-Based Services Exchange;” and in U.S. ProvisionalPatent Application 62/072,976, filed Oct. 30, 2014 and entitled“INTERCONNECTION PLATFORM FOR REAL-TIME CONFIGURATION AND MANAGEMENT OFA CLOUD-BASED SERVICES EXCHANGE;” each of which are incorporated hereinby reference in their respective entireties.

FIG. 2 is a block diagram illustrating a high-level view of a datacenter 201 that provides an operating environment for a cloud-basedservices exchange 200, according to techniques described herein.Cloud-based services exchange 200 (“cloud exchange 200”) allows acorresponding one of customer networks 204D, 204E and NSP networks204A-204C (collectively, “‘private’ or ‘carrier’ networks 204”) of anyNSPs 106A-106C or other cloud customers including customers 108A, 108Bto be directly cross-connected, via a layer 3 (L3) or layer 2 (L2)connection to any other customer network and/or to any of cloud serviceproviders 110A-110N, thereby allowing exchange of cloud service trafficamong the customer networks and CSPs 110. Data center 201 may beentirely located within a centralized area, such as a warehouse orlocalized data center complex, and provide power, cabling, security, andother services to NSPs, customers, and cloud service providers thatlocate their respective networks within the data center 201 (e.g., forco-location) and/or connect to the data center 201 by one or moreexternal links.

Network service providers 106 may each represent a network serviceprovider that is associated with a transit network by which networksubscribers of the NSP 106 may access cloud services offered by CSPs 110via the cloud exchange 200. In general, customers of CSPs 110 mayinclude network carriers, large enterprises, managed service providers(MSPs), as well as Software-as-a-Service (SaaS), Platform-aaS (PaaS),Infrastructure-aaS (IaaS), Virtualization-aaS (VaaS), and dataStorage-aaS (dSaaS) customers for such cloud-based services as areoffered by the CSPs 110 via the cloud exchange 200.

In this way, cloud exchange 200 streamlines and simplifies the processof partnering CSPs 110 and customers 108 (indirectly via NSPs 106 ordirectly) in a transparent and neutral manner. One example applicationof cloud exchange 200 is a co-location and interconnection data centerin which CSPs 110, NSPs 106 and/or customers 108 may already havenetwork presence, such as by having one or more accessible portsavailable for interconnection within the data center. This allows theparticipating carriers, customers, and CSPs to have a wide range ofinterconnectivity options in the same facility. Cloud exchange 200 ofdata center 201 includes network infrastructure 222 that provides aL2/L3 switching fabric by which CSPs 110 and customers/NSPsinterconnect. This enables an NSP/customer to have options to createmany-to-many interconnections with only a one-time hook up to theswitching network and underlying network infrastructure 222 thatpresents an interconnection platform for of cloud exchange 200. In otherwords, instead of having to establish separate connections acrosstransit networks to access different cloud service providers ordifferent cloud services of one or more cloud service providers, cloudexchange 200 allows customers to interconnect to multiple CSPs and cloudservices using network infrastructure 222 within data center 201, whichmay represent any of the edge networks described in this disclosure, atleast in part.

By being connected to and utilizing cloud exchange 200, customers canpurchase services and reach out to many end users in many differentgeographical areas without incurring the same expenses typicallyassociated with installing and maintaining multiple virtual connectionswith multiple CSPs 110. For example, NSP 106A can expand its servicesusing network 204B of NSP 106B. By connecting to cloud exchange 200, aNSP 106 may be able to generate additional revenue by offering to sellits network services to the other carriers. For example, NSP 106C canoffer the opportunity to use NSP network 204C to the other NSPs.

Cloud exchange 200 includes an programmable network platform 120 thatexposes at least one service interfaces, which may include in someexamples and are alternatively referred to herein as applicationprogramming interfaces (APIs) in that the APIs define the methods,fields, and/or other software primitives by which applications mayinvoke the programmable network platform 120. The software interfacesallow NSPs 206 and customers 108 programmable access to capabilities andassets of the cloud exchange 200. The programmable network platform 120may alternatively be referred to as a controller, provisioning platform,provisioning system, service orchestration system, etc., forestablishing end-to-end services including, e.g., connectivity betweencustomers and cloud service providers according to techniques describedherein.

On the buyer side, the software interfaces presented by the underlyinginterconnect platform provide an extensible framework that allowssoftware developers associated with the customers of cloud exchange 200(e.g., customers 108 and NSPs 206) to create software applications thatallow and leverage access to the programmable network platform 120 bywhich the applications may request that the cloud exchange 200 establishconnectivity between the customer and cloud services offered by any ofthe CSPs 110. For example, these buyer-side software interfaces mayallow customer applications for NSPs and enterprise customers, e.g., toobtain authorization to access the cloud exchange, obtain informationregarding available cloud services, obtain active ports and metro areadetails for the customer, create virtual circuits of varying bandwidthto access cloud services, including dynamic selection of bandwidth basedon a purchased cloud service to create on-demand and need based virtualcircuits to cloud service providers, delete virtual circuits, obtainactive virtual circuit information, obtain details surrounding CSPspartnered with the cloud exchange provider, obtain customized analyticsdata, validate partner access to interconnection assets, and assureservice delivery.

On the cloud service provider (seller) side, the software interfaces mayallow software developers associated with cloud providers to managetheir cloud services and to enable customers to connect to their cloudservices. For example, these seller-side software interfaces may allowcloud service provider applications to obtain authorization to accessthe cloud exchange, obtain information regarding available cloudservices, obtain active ports and metro area details for the provider,obtain active port details in a given data center for the provider,approve or reject virtual circuits of varying bandwidth created bycustomers for the purpose of accessing cloud services, obtain virtualcircuits pending addition and confirm addition of virtual circuits,obtain virtual circuits pending deletion and confirm deletion of virtualcircuits, obtain customized analytics data, validate partner access tointerconnection assets, and assure service delivery.

As further described herein, the service interface 114 facilitatesmachine-to-machine communication to enable dynamic service provisioningand service delivery assurance. In this way, the programmable networkplatform 120 enables the automation of aspects of cloud servicesprovisioning. For example, the software interfaces may provide anautomated and seamless way for customers to establish, de-install andmanage interconnection with multiple, different cloud providersparticipating in the cloud exchange. The programmable network platform120 may in various examples execute on one or virtual machines and/orreal servers of data center 201, or off-site.

In the example of FIG. 2, network infrastructure 222 represents thecloud exchange switching fabric and includes multiple ports that may bedynamically interconnected with virtual circuits by, e.g., invokingservice interface 114 of the programmable network platform 120. Each ofthe ports is associated with one of carriers 106, customers 108, andCSPs 110.

FIGS. 3A-3B are block diagrams illustrating example networkinfrastructure and service provisioning by a programmable networkplatform for a cloud exchange that aggregates the cloud services ofmultiple cloud service providers for provisioning to customers of thecloud exchange provider and aggregates access for multiple customers toone or more cloud service providers, in accordance with techniquesdescribed in this disclosure. In this example, customer networks308A-308C (collectively, “customer networks 308”), each associated witha different customer, access a cloud exchange point within a data center300 in order receive aggregated cloud services from one or more cloudservice provider networks 320, each associated with a different cloudservice provider 110. Customer networks 308 each include endpointdevices that consume cloud services provided by cloud service providernetwork 320. Example endpoint devices include servers, smart phones,television set-top boxes, workstations, laptop/tablet computers, videogaming systems, teleconferencing systems, media players, and so forth.

Customer networks 308A-308B include respective provider edge/autonomoussystem border routers (PE/ASBRs) 310A-310B. Each of PE/ASBRs 310A, 310Bmay execute exterior gateway routing protocols to peer with one of PErouters 302A-302B (“PE routers 302” or more simply “PEs 302”) over oneof access links 316A-316B (collectively, “access links 316”). In theillustrated examples, each of access links 316 represents a transit linkbetween an edge router of a customer network 308 and an edge router (orautonomous system border router) of cloud exchange point 303. Forexample, PE 310A and PE 302A may directly peer via an exterior gatewayprotocol, e.g., exterior BGP, to exchange L3 routes over access link316A and to exchange L3 data traffic between customer network 308A andcloud service provider networks 320. Access links 316 may in some casesrepresent and alternatively be referred to as attachment circuits forIP-VPNs configured in IP/MPLS fabric 301, as described in further detailbelow. Access links 316 may in some cases each include a direct physicalconnection between at least one port of a customer network 308 and atleast one port of cloud exchange point 303, with no intervening transitnetwork. Access links 316 may operate over a VLAN or a stacked VLAN(e.g, QinQ), a VxLAN, an LSP, a GRE tunnel, or other type of tunnel.

While illustrated and primarily described with respect to L3connectivity, PE routers 302 may additionally offer, via access links316, L2 connectivity between customer networks 308 and cloud serviceprovider networks 320. For example, a port of PE router 302A may beconfigured with an L2 interface that provides, to customer network 308A,L2 connectivity to cloud service provider 320A via access link 316A,with the cloud service provider 320A router 312A coupled to a port of PErouter 304A that is also configured with an L2 interface. The port of PErouter 302A may be additionally configured with an L3 interface thatprovides, to customer network 308A, L3 connectivity to cloud serviceprovider 320B via access links 316A. PE 302A may be configured withmultiple L2 and/or L3 sub-interfaces such that customer 308A may beprovided, by the cloud exchange provider, with a one-to-many connectionto multiple cloud service providers 320.

To create an L2 interconnection between a customer network 308 and acloud service provider network 320, in some examples, IP/MPLS fabric 301is configured with an L2 bridge domain (e.g., an L2 virtual privatenetwork (L2VPN) such as a virtual private LAN service (VPLS), E-LINE, orE-LAN) to bridge L2 traffic between a customer-facing port of PEs 302and a CSP-facing port of cloud service providers 320. In some cases, acloud service provider 320 and customer 308 may have access links to thesame PE router 302, 304, which bridges the L2 traffic using the bridgedomain.

To create an L3 interconnection between a customer network 308 and acloud service provider network 320, in some examples, IP/MPLS fabric 301is configured with a L3 virtual routing and forwarding instances (VRFs),as described in further detail below with respect to FIG. 4.

Each of access links 316 and aggregation links 322 may include a networkinterface device (NID) that connects customer network 308 or cloudservice provider 328 to a network link between the NID and one of PErouters 302, 304. Each of access links 316 and aggregation links 322 mayrepresent or include any of a number of different types of links thatprovide L2 and/or L3 connectivity.

In this example, customer network 308C is not an autonomous systemhaving an autonomous system number. Customer network 308C may representan enterprise, network service provider, or other customer network thatis within the routing footprint of the cloud exchange point. Customernetwork includes a customer edge (CE) device 311 that may executeexterior gateway routing protocols to peer with PE router 302B overaccess link 316C. In various examples, any of PEs 310A-310B mayalternatively be or otherwise represent CE devices.

Access links 316 include physical links. PE/ASBRs 310A-310B, CE device311, and PE routers 302A-302B exchange L2/L3 packets via access links316. In this respect, access links 316 constitute transport links forcloud access via cloud exchange point 303. Cloud exchange point 303 mayrepresent an example of any of cloud exchange points 128. Data center300 may represent an example of data center 201.

Cloud exchange point 303, in some examples, aggregates customers 308access to the cloud exchange point 303 and thence to any one or morecloud service providers 320. FIGS. 3A-3B, e.g., illustrate access links316A-316B connecting respective customer networks 308A-308B to PE router302A of cloud exchange point 303 and access link 316C connectingcustomer network 308C to PE router 302B. Any one or more of PE routers302, 304 may comprise ASBRs. PE routers 302, 304 and IP/MPLS fabric 301may be configured according to techniques described herein tointerconnect any of access links 316 to any of cloud aggregation links322. As a result, cloud service provider network 320A, e.g., needs onlyto have configured a single cloud aggregate link (here, access link322A) in order to provide services to multiple customer networks 308.That is, the cloud service provider operating cloud service providernetwork 302A does not need to provision and configure separate servicelinks from cloud service provider network 302A to each of PE routers310, 311, for instance, in order to provide services to each of customernetwork 308. Cloud exchange point 303 may instead cross-connect cloudaggregation link 322A and PE 312A of cloud service provider network 320Ato multiple cloud access links 316 to provide layer 3 peering andnetwork reachability for the cloud services delivery.

In addition, a single customer network, e.g., customer network 308A,need only to have configured a single cloud access link (here, accesslink 316A) to the cloud exchange point 303 within data center 300 inorder to obtain services from multiple cloud service provider networks320 offering cloud services via the cloud exchange point 303. That is,the customer or network service provider operating customer network 308Adoes not need to provision and configure separate service linksconnecting customer network 308A to different PE routers 312, forinstance, in order to obtain services from multiple cloud serviceprovider networks 320. Cloud exchange point 303 may insteadcross-connect cloud access link 316A (again, as one example) to multiplecloud aggregate links 322 to provide layer 3 peering and networkreachability for the cloud services delivery to customer network 308A.

Cloud service provider networks 320 each includes servers configured toprovide one or more cloud services to users. These services may becategorized according to service types, which may include for examples,applications/software, platforms, infrastructure, virtualization, andservers and data storage. Example cloud services may includecontent/media delivery, cloud-based storage, cloud computing, onlinegaming, IT services, etc.

Cloud service provider networks 320 include PE routers 312A-312D thateach executes an exterior gateway routing protocol, e.g., eBGP, toexchange routes with PE routers 304A-304B (collectively, “PE routers304”) of cloud exchange point 303. Each of cloud service providernetworks 320 may represent a public, private, or hybrid cloud. Each ofcloud service provider networks 320 may have an assigned autonomoussystem number or be part of the autonomous system footprint of cloudexchange point 303.

In the illustrated example, an Internet Protocol/Multiprotocol labelswitching (IP/MPLS) fabric 301 interconnects PEs 302 and PEs 304.IP/MPLS fabric 301 include one or more switching and routing devices,including PEs 302, 304, that provide IP/MPLS switching and routing of IPpackets to form an IP backbone. In some example, IP/MPLS fabric 301 mayimplement one or more different tunneling protocols (i.e., other thanMPLS) to route traffic among PE routers and/or associate the trafficwith different IP-VPNs. In accordance with techniques described herein,IP/MPLS fabric 301 implement IP virtual private networks (IP-VPNs) toconnect any of customers 308 with multiple cloud service providernetworks 320 to provide a data center-based ‘transport’ and layer 3cross-connect. Whereas service provider-based IP backbone networksrequire wide-area network (WAN) connections with limited bandwidth totransport service traffic from layer 3 services providers to customers,the cloud exchange point 303 as described herein ‘transports’ servicetraffic and cross-connects cloud service providers 320 to customers 308within the high-bandwidth local environment of data center 300 providedby a data center-based IP/MPLS fabric 301. In some examples, IP/MPLSfabric 301 implements IP-VPNs using techniques described in Rosen &Rekhter, “BGP/MPLS IP Virtual Private Networks (VPNs),” Request forComments 4364, February 2006, Internet Engineering Task Force (IETF)Network Working Group, the entire contents of which is incorporated byreference herein. In some example configurations, a customer network 308and cloud service provider network 320 may connect via respective linksto the same PE router of IP/MPLS fabric 301.

Access links 316 and aggregation links 322 may include attachmentcircuits that associate traffic, exchanged with the connected customernetwork 308 or cloud service provider network 320, with virtual routingand forwarding instances (VRFs) configured in PEs 302, 304 andcorresponding to IP-VPNs operating over IP/MPLS fabric 301. For example,PE 302A may exchange IP packets with PE 310A on a bidirectionallabel-switched path (LSP) operating over access link 316A, the LSP beingan attachment circuit for a VRF configured in PE 302A. As anotherexample, PE 304A may exchange IP packets with PE 312A on a bidirectionallabel-switched path (LSP) operating over access link 322A, the LSP beingan attachment circuit for a VRF configured in PE 304A. Each VRF mayinclude or represent a different routing and forwarding table withdistinct routes.

PE routers 302, 304 of IP/MPLS fabric 301 may be configured inrespective hub-and-spoke arrangements for cloud services, with PEs 304implementing cloud service hubs and PEs 302 being configured as spokesof the hubs (for various hub-and-spoke instances/arrangements). Ahub-and-spoke arrangement ensures that service traffic is enabled toflow between a hub PE and any of the spoke PEs, but not directly betweendifferent spoke PEs. As described further below, in a hub-and-spokearrangement for data center-based IP/MPLS fabric 301 and for southboundservice traffic (i.e., from a CSP to a customer) PEs 302 advertiseroutes, received from PEs 310, to PEs 304, which advertise the routes toPEs 312. For northbound service traffic (i.e., from a customer to aCSP), PEs 304 advertise routes, received from PEs 312, to PEs 302, whichadvertise the routes to PEs 310.

For some customers of cloud exchange point 303, the cloud exchange point303 provider may configure a full mesh arrangement whereby a set of PEs302, 304 each couple to a different customer site network for thecustomer. In such cases, the IP/MPLS fabric 301 implements a layer 3 VPN(L3VPN) for cage-to-cage or redundancy traffic (also known as east-westor horizontal traffic). The L3VPN may effectuate a closed user groupwhereby each customer site network can send traffic to one another butcannot send or receive traffic outside of the L3VPN.

PE routers may couple to one another according to a peer model withoutuse of overlay networks. That is, PEs 310 and PEs 312 may not peerdirectly with one another to exchange routes, but rather indirectlyexchange routes via IP/MPLS fabric 301. In the example of FIG. 3B, cloudexchange point 303 is configured to implement multiple layer 3 virtualcircuits 330A-330C (collectively, “virtual circuits 330”) tointerconnect customer network 308 and cloud service provider networks322 with end-to-end IP paths. Each of cloud service providers 320 andcustomers 308 may be an endpoint for multiple virtual circuits 330, withmultiple virtual circuits 330 traversing one or more attachment circuitsbetween a PE/PE or PE/CE pair for the IP/MPLS fabric 301 and theCSP/customer. A virtual circuit 330 represents a layer 3 path throughIP/MPLS fabric 301 between an attachment circuit connecting a customernetwork to the fabric 301 and an attachment circuit connecting a cloudservice provider network to the fabric 301. Each virtual circuit 330 mayinclude at least one tunnel (e.g., an LSP and/or Generic RouteEncapsulation (GRE) tunnel) having endpoints at PEs 302, 304. PEs 302,304 may establish a full mesh of tunnels interconnecting one another.

Each virtual circuit 330 may include a different hub-and-spoke networkconfigured in IP/MPLS network 301 having PE routers 302, 304 exchangingroutes using a full or partial mesh of border gateway protocol peeringsessions, in this example a full mesh of Multiprotocol Interior BorderGateway Protocol (MP-iBGP) peering sessions. MP-iBGP or simply MP-BGP isan example of a protocol by which routers exchange labeled routes toimplement MPLS-based VPNs. However, PEs 302, 304 may exchange routes toimplement IP-VPNs using other techniques and/or protocols.

In the example of virtual circuit 330A, PE router 312A of cloud serviceprovider network 320A may send a route for cloud service providernetwork 320A to PE 304A via a routing protocol (e.g., eBGP) peeringconnection with PE 304A. PE 304A associates the route with ahub-and-spoke network, which may have an associated VRF, that includesspoke PE router 302A. PE 304A then exports the route to PE router 302A;PE router 304A may export the route specifying PE router 304A as thenext hop router, along with a label identifying the hub-and-spokenetwork. PE router 302A sends the route to PE router 310B via a routingprotocol connection with PE 310B. PE router 302A may send the routeafter adding an autonomous system number of the cloud exchange point 303(e.g., to a BGP autonomous system path (AS_PATH) attribute) andspecifying PE router 302A as the next hop router. Cloud exchange point303 is thus an autonomous system “hop” in the path of the autonomoussystems from customers 308 to cloud service providers 320 (andvice-versa), even though the cloud exchange point 303 may be basedwithin a data center. PE router 310B installs the route to a routingdatabase, such as a BGP routing information base (RIB) to provide layer3 reachability to cloud service provider network 320A. In this way,cloud exchange point 303 “leaks” routes from cloud service providernetworks 320 to customer networks 308, without cloud service providernetworks 320 to customer networks 308 requiring a direct layer peeringconnection.

PE routers 310B, 302A, 304A, and 312A may perform a similar operation inthe reverse direction to forward routes originated by customer network308B to PE 312A and thus provide connectivity from cloud serviceprovider network 320A to customer network 308B. In the example ofvirtual circuit 330B, PE routers 312B, 304A, 302A, and 310B exchangeroutes for customer network 308B and cloud service provider 320B in amanner similar to that described above for establishing virtual circuit330B. As a result, cloud exchange point 303 within data center 300internalizes the peering connections that would otherwise be establishedbetween PE 310B and each of PEs 312A, 312B so as to perform cloudaggregation for multiple layer 3 cloud services provided by differentcloud service provider networks 320A, 320B and deliver the multiple,aggregated layer 3 cloud services to a customer network 308B having asingle access link 316B to the cloud exchange point 303. Absent thetechniques described herein, fully interconnecting customer networks 308and cloud service provider networks 320 would require 3×3 peeringconnections between each of PEs 310 and at least one of PEs 312 for eachof cloud service provider networks 320. For instance, PE 310A wouldrequire a layer 3 peering connection with each of PEs 312. With thetechniques described herein, cloud exchange point 303 may fullyinterconnect customer networks 308 and cloud service provider networks320 with one peering connection per site PE (i.e., for each of PEs 310and PEs 312) by internalizing the layer 3 peering and providing datacenter-based ‘transport’ between cloud access and cloud aggregateinterfaces.

In examples in which IP/MPLS fabric 301 implements BGP/MPLS IP VPNs orother IP-VPNs that use route targets to control route distributionwithin the IP backbone, PEs 304 may be configured to import routes fromPEs 302 and to export routes received from PEs 312, using differentasymmetric route targets. Likewise, PEs 302 may be configured to importroutes from PEs 304 and to export routes received from PEs 310 using theasymmetric route targets. Thus, PEs 302, 304 may configured to implementadvanced L3VPNs that each includes a basic backbone L3VPN of IP/MPLSfabric 301 together with extranets of any of customer networks 308 andany of cloud service provider networks 320 attached to the basicbackbone L3VPN. Each advanced L3VPN constitutes a cloud service deliverynetwork from a cloud service provider network 320 to one or morecustomer networks 308, and vice-versa. In this way, cloud exchange point303 enables any cloud service provider network 320 to exchange cloudservice traffic with any customer network 308 while internalizing thelayer 3 routing protocol peering connections that would otherwise beestablished between pairs of customer networks 308 and cloud serviceprovider networks 320 for any cloud service connection between a givenpair. In other words, the cloud exchange point 303 allows each ofcustomer networks 308 and cloud service provider networks 320 toestablish a single (or more for redundancy or other reasons) layer 3routing protocol peering connection to the data center-based layer 3cross-connect. By filtering routes from cloud service provider networks320 to customer networks 308, and vice-versa, PEs 302, 304 therebycontrol the establishment of virtual circuits 330 and the flow ofassociated cloud service traffic between customer networks 308 and cloudservice provider networks 320 within a data center 300. Routesdistributed into MP-iBGP mesh 318 may be VPN-IPv4 routes and beassociated with route distinguishers to distinguish routes fromdifferent sites having overlapping address spaces.

Programmable network platform 120 may receive service requests forcreating, reading, updating, and/or deleting end-to-end services of thecloud exchange point 303. In response, programmable network platform 120may configure PEs 302, 304 and/or other network infrastructure ofIP/MPLS fabric 301 to provision or obtain performance or otheroperations information regarding the service. Operations forprovisioning a service and performed by programmable network platform120 may include configuring or updating VRFs, installing SDN forwardinginformation, configuring LSPs or other tunnels, configuring BGP,configuring access links 316 and aggregation links 322, or otherwisemodifying the configuration of the IP/MPLS fabric 301. Other operationsmay include making service requests to an orchestration system for cloudservice provider networks 320, as described in further detail below.

FIG. 4 is a block diagram illustrating an example of a data center-basedcloud exchange point in which routers of the cloud exchange point areconfigured by programmable network platform 120 with VPN routing andforwarding instances for routing and forwarding aggregated servicetraffic from multiple cloud service provider networks to a customernetwork, according to techniques described herein. In this example, toestablish virtual circuits 330A-330B, PE routers 302A and 304A ofIP/MPLS fabric 301 are configured with VRFs. PE 302A is configured withVRFs 402A and 404A, while PE 304A is configured with VRFs 402B and 404B.VRF 402A is configured to import routes exported by VRF 402B, and VRF402B is configured to import routes exported by VRF 402A. Theconfiguration may include asymmetric route targets for import/exportbetween VRFs 402A, 402B. VRF 404A is configured to import routesexported by VRF 402B, and VRF 402B is configured to import routesexported by VRF 402A. The configuration may include asymmetric routetargets for import/export between VRFs 402A, 402B. This configurationwhereby a customer is able to access multiple layer 3 services fromdifferent CSPs each associated with separate VRFs to access the layer 3services provides isolation of respective traffic exchanged with theCSPs. In some examples, PE 302A may be configured with a single VRF toimport routes exported by both VRF 402B and VRF 404B. As noted abovewith respect to FIGS. 3A-3B, PEs 302, 304 may be further configured tobridge layer 2 traffic between customer 308B and cloud service providers320.

In this example, PE 304A operates BGP or other route distributionprotocol peering connections 406B, 408B with respective PEs 312A, 312Bto exchange routes with respective cloud service provider networks 320A,320B. PE 302A operates a BGP or other route distribution protocolpeering connection 410 with PE 310B to exchange routes with customernetwork 308B. In some examples, PEs 302A, 304A may be staticallyconfigured with routes for the site networks.

An administrator or a programmable network platform described herein forcloud exchange point 303 may configure PEs 302A, 304A with the VRF402A-402B, 404A-404B in order to leak routes between PEs 312 and PE 310Band facilitate layer 3 connectivity for end-to-end IP paths illustratedhere by virtual circuits 330, while potentially optimizing theend-to-end IP paths by fostering data center-based or at leastmetro-based connectivity. Cloud exchange point 303 may thus providededicated cloud service provider access to customer network 308B by wayof private and/or public routes for the cloud service provider networks320. In the northbound direction, cloud exchange point 303 may providededicated cloud service provider distribution to multiple customernetworks 308 by way of private and/or public routes for the customernetworks 308. Neither PE 310B nor any of PEs 302A, 304A need access tothe full Internet BGP routing table in order to reach cloud serviceprovider networks 320 or customer networks 308. Moreover, PEs 302A, 304Amay be configured to aggregate customer/CSP routes and/or servicetraffic based on any one or more of physical, IP, service, and VRFs.

FIG. 5 is a block diagram illustrating a platform for a softwarecontrolled network, the platform operating in accordance with one ormore techniques of the present disclosure. FIG. 5 illustrates aprogrammable network platform 10000 that includes multiple components,which collectively provide for dynamic configuration and management of acloud-based services exchange, or “cloud exchange.” These components mayprovide virtual connections for cloud services delivery from multiplecloud service providers to one or more cloud customers. Programmablenetwork platform 10000 includes centralized network control (CNC) system10002, one or more network field units (NFUs) 10004, software-definednetworking (SDN) controller 10006, hardware configurators 10008,infrastructure data collectors 10010, and information technology systems(10010).

Programmable network platform 10000 may provide for the orchestration ofa service across multiple service providers and allow one of the serviceproviders to be the service owner in terms of the service monitoring,assurance and billing. Programmable network platform 10000 may providethe process and apparatus for multiple service provider orchestrationsystem to securely communicate with each other to deliver a combinedservice on demand in a single click manner. Programmable networkplatform 10000 may represent an example instance of programmable networkplatform 120 or another programmable network platform, controller, orsystem described herein for provisioning services and assuring servicedelivery.

In the example of FIG. 5, CNC system 10002 enables the automation ofaspects of cloud services provisioning. As such, CNC system 10002 mayprovide one or more software interfaces that allow customers toestablish, de-install and manage interconnections with multiple,different cloud providers participating in the cloud exchange in anautomated and seamless manner. CNC system 10002 may include logic toreceive a business service request via an API call and convert that intothe necessary business instantiation parameters and network provisioningparameters to be delivered and assured as a business service. CNC system10002 may be the central intelligent processing unit of theorchestration system (e.g., programmable network platform 10000) andthere may be one logical instance of this intelligent logic present perinstantiation. CNC system 10002 also has the capability of communicatingwith a third party orchestration system if needed by the servicerequest. CNC system 10002 may provide service assurance using a Monitor,Analyze, Plan and Execute (MAPE) loop methodology, as further discussedin this disclosure, and is implemented to ensure the service levelagreements are adhered to by the service.

In some examples, NFU 10004 is implemented as a self-contained unit thatreceives requests or instructions from CNC system 10002 to configurenetwork infrastructure of a cloud exchange point for one or moreservices. For instance, NFU 10004 may comprise a combination of hardwareand software. In some examples, NFU 10004 may be a virtual machine. Inany case, NFU 10004 receives requests or instructions CNC system 10002based on customer requests submitted to CNC system 10002. As furtherdescribed below, NFU 10004 may determine whether sufficient resourcesexist to provide the services requested by CNC system 10002. Ifsufficient resources exist, NFU 10004 may communicate or otherwiseinteroperate with SDN controller 10006, hardware configurators 10008,and infrastructure data collectors 10010 to configure the networkinfrastructure to provide the requested service. NFU 10004 may representa globally distributed intelligent logical unit that receives networkinstantiation commands from CNC system 10002 and instantiates andconfigures the network resource that is needed to deliver the service.NFU 10004 may have the intelligence to deliver and assure networkservices as per the request of CNC system 10002. NFU 10004 may have itsown MAPE loop (e.g., as shown in FIG. 9) to ensure that the networkservices delivered by the unit is assured for the life cycle of theservice.

In some examples, multiple cloud exchange points may be geographicallydispersed. Each geographically positioned cloud exchange point may havea corresponding NFU that is geographically positioned at the samelocation as the respective cloud exchange point. The corresponding NFUmay configure and otherwise manage the network infrastructure of theparticular geographically-positioned cloud exchange point. In this way,a particular NFU may receive requests or instructions from CNC system10002 and configure the network infrastructure of the cloud exchangepoint that is managed by the particular NFU. In some cases, multiplecloud exchange points of a metropolitan area make up a metro-based cloudexchange managed by a single NFU.

NFU 10004 may therefore represent the distributed processing unit ofprogrammable network platform 10000, which provides programmable networkplatform 10000 with the ability to horizontal scale and deliver andassure services. NFU 10004 is the component of programmable networkplatform 10000 that may provide the functionality of delivering theservices in a vendor agnostic and form factor agnostic manner. As shownin FIG. 5, NFU 10004 has several software components that enable thedistributed processing unit to deliver the services.

In order to provision services and virtual connections to cloudcustomers and cloud service providers, CNC system 10002 includes aservice selector 10012. In some examples, service selector 10012 mayoperate as an API gateway. For example, service selector 10012 mayexpose software interfaces defined according to one or more APIs.Requests and/or instructions received by service selector 10012 may beinclude the form of create, read, update, and/or delete (CRUD) requestsmade with respect to services provided by and/or delivered by the cloudexchange. Applications may invoke endpoints of the APIs provided byservice selector 10012, which may in turn invoke service provisioningengine 10014. Service selector 10012 may execute on one or virtualmachines and/or real servers, for instance. Although shown as a singleelement in FIG. 5, service selector 10012 may comprise a cluster of oneor more physical and/or virtual computing machines executing on one ormore physical processors. In some aspects, service selector 10012provides a service catalog that describes available services andproviders for the available services.

Service provisioning engine 10014 may receive requests to provisionservices from service selector 10012. Service provisioning engine 10014,in conjunction with network field unit 10004, organizes, directs andintegrates underlying hardware and software sub-systems for managingvarious aspects of service provisioning within the networkinfrastructure as well as cloud services management. For instance,service provisioning engine 10014 may provide a rule-driven workflowengine that operates between service selector 10012 and the underlyinginterconnect platform of a cloud exchange that is configured by networkfield unit 10004. In this way, service provisioning engine 10014 can beinvoked via service selector 10012 by customer-proprietary applications,a cloud provider-based customer portal, and/or cloud service providersystems, for direct participation with the programmable network platformof a cloud exchange network infrastructure that is configured by networkfield unit 10004. As described in further detail below with respect toFIG. 6 et al., service provisioning engine 10014 may include athird-party service connector that communicates with the third partyorchestration systems to ensure that the service is adequately networkedtogether to provide the end-to-end cloud-based service fulfillment. Asfurther described below, NFU 10004 may receive instructions and/orrequests from CNC system 10002, which NFU 10004 uses to provisionservices at one or more cloud exchange points.

Service provisioning engine 10014 may query and store service telemetryand analytics data (STAD) 10016 in one or more data stores. STAD 10016may include metrics about the quantity, type, definition, and consumersof services that are configured by service provisioning engine 10014.STAD 10016 may include analytics information based on raw metrics datafrom NFU 10004. For instances, analysis information of STAD 10016 mayinclude historical statistics and/or real-time statistics, which may beanalyzed on various dimensions, such as consumer, service type, serviceuse, to name only a few examples.

CNC system 10002 may also include financial logic 10018. Financial logic10018 may store accounting information for customers. For instance,financial logic 10018 may store billing information for customers, suchas name, address, phone number, email, to name only a few examples. Whenservice provisioning engine 10014 configures a service for a customerthat includes a service charge, financial logic 10018 may store suchexpense information. In this way, financial logic 10018 may provide anaccounting of services purchased by a customer and provide billing forsuch services.

CNC system 10002 may include Information Technology (IT) gateway 10020that interfaces with IT systems 100022. IT systems 100022 may includeone or more computing devices, such as desktop computers, tablets,smartphones, and servers, to name only a few examples. IT systems 100022may provide one or more user interfaces to administrators, which may useIT systems 100022 to administer CNC system 10002. IT systems 100022 may,for example, receive user inputs to configure CNC system 10002 and/orNFU 10004. Based on the user inputs, IT systems 100022 may send requestsand/or instructions to CNC system 10002, which are received by ITgateway 10020. In some examples, CNC system 10002 may provide orotherwise expose one or more RESTful interfaces that can be called orotherwise invoked by IT systems 100022. IT gateway 10020 may route suchinstructions or requests to other components within CNC system 10002 forfurther processing based on the type of requests and/or instructions.

As described above, NFU 10004 may receive requests or instructions fromCNC system 10002 to provision one or more services. Network provisioningengine 10024 may receive the requests and/or instructions from serviceprovisioning engine 10014. Network provisioning engine 10024 maydetermine whether sufficient resources exist to satisfy a request for aservice to be configured at a cloud exchange point. In some examples,network provisioning engine 10024 may query one or more components suchas SDN controller 10006, hardware configurators 10008, and/or networktelemetry and analytics data (NTAD) 10026. If sufficient resources existto provision a requested service, network provisioning engine 10024 maysend instructions and/or requests to one or more of SDN controller 10006and/or hardware configurators 10008 that cause each respective componentto be configured to provision the requested service. As such, networkprovisioning engine 10024 provides the functionality of selecting thevendor, and form factor in which the service is delivered. Networkprovisioning engine 10024 also provides the policy manager functionalityto ensure the service is delivered in the correct order of operations.

In some examples, network provisioning engine 10024 of NFU 10004 mayinclude a Network Appliance Sizing Engine (not shown) that provides thefunctionality of ensuring the network appliance is properly sized forthe appropriate SLA to be delivered by the appliance. In some examples,NFU 10004 may include a Device Selection and Handler (not shown) thatprovides the functionality of selecting the correct device to deliverthe service, and convert the network commands to the appropriateconfiguration commands for the selected device. For example NFU 10014may access a list that describes the capabilities of virtual and/ordedicated appliances within the cloud exchange for providing nativeservices, such as firewall (FW), network address translation (NAT), anddeep-packet inspection (DPI), to service traffic traversing the cloudexchange. NFU 10004 may select a device from the list to satisfy theservice request, as described in further detail below with respect toFIG. 8, for instance.

Network provisioning engine 10024 may query and store network telemetryand analytics data (NTAD) 10026 in one or more data stores. NTAD 10026may include metrics about the quantity, type, definition, of network andresource configurations that are configured by NFU 10004. NTAD 10026 mayinclude analytics information from infrastructure data collectors 10010based on raw metrics data for resources used in a particular service.For instances, analysis information of NTAD 10026 may include historicalstatistics and/or real-time statistics.

As shown in FIG. 5, one or more SDN controllers 10006 may configurenetwork resources, such as routers, switches, bridges, and the like,which provide the physical infrastructure to carry network trafficthrough a cloud exchange point. One or more hardware configurators 10008may configure hardware resources, such as servers or the above-mentionednetwork resources; resources within servers and network resourcesincluding processor allocation, memory allocation; storage appliances;other hardware resources; and software configurations that may beconfigured to provision services to a customer. One or moreinfrastructure data collectors 10010 may collect metrics about thequantity, type, definition, of network and resource configurations thatare configured by NFU 10004. For instance, infrastructure datacollectors 10010 may monitor and measure metrics of network resourcesand any other resources configured to provision services to a customer.Infrastructure data collectors 10010 may store such metrics in NTAD10026.

NFU 10004 and CNC system 10002 may include network assurance engine10028 and service assurance engine 10030, respectively. Networkassurance engine 10028 may determine, based on NTAD 10026, whetherinfrastructure configured to provide services is providing asatisfactory level of service. For example, outages, resourceconsumption overages, hardware and/or software failures or problems, andother events may affect the quality of services provided by the networkinfrastructure at a cloud exchange point. Network assurance engine 10028may monitor NTAD 10026, and in some cases, send information to serviceassurance engine 10030. In some examples, the information may includealerts if service levels are not being met, or more specifically alertsfor outages, resource consumption overages, hardware and/or softwarefailures or problems. In some examples, information sent by networkassurance engine 10028 to service assurance engine 10030 may beinformational rather than based on a specific event. For instance,network assurance engine 10028 may send information about theperformance of infrastructure to service assurance engine on aparticular schedule or interval, and/or on continuous or real-timebasis. In some examples NTAD 10026 may contain a set of structuredand/or unstructured databases that enable the service provisioningengine 10014 and network assurance engine 10028 to appropriately storeand retrieve data to support the operation of programmable networkplatform 10000.

Network assurance engine 10028 may provide the functionality of assuringthe network configuration created is assured as per the networking SLAsrequested by CNC system 10002. The Network Assurance Engine is comprisedof several sub-components that deliver the assurance through a MAPE loopincluding: (1) Monitoring, which is performed by several data collectorsthat are programmed to monitor and gather data for a given service; (2)Analyzing, which analyzes the data collected by the data collectors tocompare and ensure that the services are compliant with the requestedSLAs (3) Planning, which in the event a service or set of services areout of compliance, a planning module will make the decisions if thecurrent non-compliance can be mitigated locally or needs to escalated tothe CNC system 10002 for further processing; and (4) Executing, which isbased on the decisions taken by the planning module to execute actionsin the event the non-compliance can be locally mitigated.

Service assurance engine 10030 may receive information from networkassurance engine 10028 and may compare the information with servicelevel information, such as service level agreements, included in STAD10016. By comparing information about the performance of infrastructurewith service level information in STAD 10016, service assurance enginemay send service level information to customers using one or moreservice assurance APIs, and whether such service level agreements arebeing met. In this way, customers may monitor or otherwise evaluate thequality of service provided by one or more cloud exchange points.

As described above, programmable network platform 10000 may bridgebusiness systems, such as customers and cloud service providers, withoperations systems, such as the network infrastructure of one or morecloud exchange points to improve operational efficiency. As such,programmable network platform 10000 may provide improved visibility tomonitor and assure the end-to-end service and its components.Accordingly, programmable network platform 10000, unlike conventionalsystems may include the capability to perform the provisioning andassurance of services across multiple orchestration systems for multiplecloud providers. CNC system 10002 may operate as the master controllerthat performs the function of receiving a service request thatencapsulates the business requirements for the service, and usingbusiness, network and partner sub-system logic to instantiate and assurethe service. As shown in FIG. 5, CNC system 10002 is made up of multipledifferent software modules performing different functions of fulfillinga service request. Programmable network platform 10000 may provide adistributed orchestration system for creating services and distributingthe intelligence of delivering and assuring services. Additionally,programmable network platform 10000 may provide a distributed systemthat is able to communicate with third party service orchestrationsystems and deliver a distributed service, as described in furtherdetail below with respect to FIG. 6.

Programmable network platform 10000 may provide service orchestration ofa business level service across heterogeneous service providers. Thedefinition of the service policy, quality, service level agreements andcost as a coordinated service topology may be provided at programmablenetwork platform 10000. Programmable network platform 10000 may definethe individual sub-component level topology, policy, SLA and cost interms of specification and enforcement.

Programmable network platform 10000 is an intelligent centralizedservice delivery and assurance system with the ability to have faultmitigation Monitor/Analyze/Plane/Execute (MAPE) loop, as shown in FIGS.7 and 9 that will ensure the service delivered by the system is assuredto adhere the service level agreement for the life cycle of the service.Programmable network platform 10000 not only delivers services that canbe offered by its own delivery infrastructure but also has thecapability to communicate across other third-party orchestration systemsto deliver a combined homogeneous service. Programmable network platform10000, or more specifically CNC system 10002, may be the central controlcenter for both operations and business related functions to beperformed.

NFU 10004 and CNC system 10002 may also fulfill the need for having adistributed orchestration system for creating services and distributingthe intelligence of delivering and assuring service. Additionally, NFU10004 and CNC system 10002 may fulfill the need for the distributedsystem to be able to communicate with third party service orchestrationsystems to deliver a distributed service. Programmable network platform10000 provides the advantage of providing a distributed, horizontallyscaling architecture. CNC 10002 and one or more NFUs 10004 may providethe functionality of delivering and assuring a business service into twodistinctly separate functions, (1) CNC—may handle the function ofconverting the business request into service parameters, (2) NFU—mayhandle the function of converting the service parameters into networkparameters and instantiating the service.

FIG. 6 is a block diagram illustrating service provisioning engine 10014of FIG. 5 in further detail, in accordance with one or more techniquesof the present disclosure. As shown in FIG. 6, service provisioningengine 10014 may include a service policy manager 10400. Service policymanager 10400 may receive service requests and/or instructions fromother components of CNC system 10002, such as service selector 10012,service assurance engine 10030, financial logic 10018 and IT gateway10020. Requests and/or instructions received by service policy manager10400 may take the form of create, read, update, and/or delete (CRUD)requests. Service policy manager 10400 may provide a set of APIs thatallow other components of CNC system 1000 to send service requestsand/or instructions to service policy manager 10400. Service policymanager 10400 may direct such service requests and/or instructions toother components within service policy manager 10400.

Service provisioning engine 10014 may include native services 10402 andthird-party services orchestrated via third-party orchestration modules10404. Native services 10402 may include services designed and/orimplemented by an operator of CNC system 10002. For instance, nativeservices may be used to configure virtual circuits at one or more cloudexchange points. Examples of native services 10402 may include but arenot limited to a port service 10402A, one or more layer 3 (L3)connectivity services 10402B, one or more layer 2 (L2) connectivityservices 10402C, and one or more connectivity services provided in anOSI layer that is greater than L3 such as Application, Presentation,Session, and Transport layer services (“L3+ services 10402D”). Portservice 10402A may identify and/or configure one or more ports toprovide one or more services at a cloud exchange point. L2, L3 and L3+services may refer to the OSI or TCP/IP layer at which a particularservice is applied.

In some examples, a programmable network platform described herein mayprovide for orchestrating a service that involves both native andthird-party components as single service while ensuring policy,security, and service level agreement (SLA) consistency. Theprogrammable network platform may orchestrate the third-party servicecomponents using a third-party (or “partner”) orchestration module (or“plugin”). A third-party orchestration module allows a third-partyorchestration system to register its capabilities (e.g., servicecatalog, policy, security and SLA) with the programmable networkplatform. The cloud service provider, as the service owner, may use theprogrammable network platform to direct the third-party orchestrationsystem, via the corresponding third-party orchestration module, as partof the workflow for the service delivery to stand-up and deliver athird-party service for the service.

As a result, the programmable network platform may be adapted andextended by registering (or updating) third-party orchestration modulesfor any third-party orchestration system. This may allow the cloudservice provider that administers the programmable network platform toprovide interconnectivity between customers and cloud service providersto also broker and delivery layer 3 services of the cloud serviceproviders to the customers.

As illustrated in FIG. 6, service provisioning engine 10014 may includeone or more third-party orchestration modules 10404 to enableorchestration of cloud services by the service provisioning engine10014. In some examples, third-party orchestration modules 10404 may bedesigned and/or implemented by respective cloud service providers, otherthan administrators of a cloud-exchange point. Although designed andimplemented by third parties, third-party orchestration modules 10404are hosted and executed at a cloud exchange point. In this way, thirdparties may design and implement third-party orchestration modules 10404that are hosted and executed at a cloud exchange point.

Each of third-party orchestration modules 10404 may present a commoninterface to the service provisioning engine 10014 for requesting cloudservices from the cloud service providers. The interface may include acatalog interface by which a cloud service provider can publish its listof available cloud services, together with available policy, security,SLA parameters, and costs for the cloud services, to the programmablenetwork platform. The programmable network platform may replicate thelist of available cloud services for the various cloud service providersto customers via a customer portal. For example, cloud service providersfor respective third-party orchestration modules 10404 may each offer adata storage service and publish this offering via the third-partyorchestration modules 10404. In turn, the programmable network platformwith which a given third-party orchestration module 10404 has registeredmay invoke the common interface to request orchestration of one of theoffered cloud services. The third-party orchestration module 10404responsively communicates with an orchestration system of thecorresponding cloud service provider to cause the cloud service providernetwork to set up the requested layer 3 cloud service according toservice parameters in the request. Such service parameters may includepolicy, service-specific information specific to the type of layer 3cloud service (e.g., a data storage size for a dSaaS service),connectivity information (e.g., L3 address for a customer or anothercloud service provider network), QoS information, among otherparameters.

Upon instantiation of a cloud service requested by the serviceprovisioning engine via a third-party orchestration module 10404A (forinstance), the third-party orchestration module 10404A may receiveconnectivity information that enables the cloud exchange to connect tothe instantiated cloud service. This connectivity information, or“network handle,” may include, e.g., a layer 3 route to the service; aVxLAN, VLAN, or other tunnel identifier usable by a cloud serviceprovider network-facing cloud exchange router for accessing the cloudservice (for instance, forwarding service traffic to the cloud serviceor identifying service traffic for the cloud service received by thecloud exchange point from the cloud service provider network). Thethird-party orchestration module 10404A provides the connectivityinformation 1405 to the service provisioning engine 10014 (e.g., via thecommon interface), which uses the connectivity information and STAD10406 to procure service data and eventually to generate the networkprovisioning data at 10410. STAD 10406 provides service provisioningengine 10014 with indications of services provided. In this way, thecloud exchange provider is absolved from having to establish a cloudservice via a corresponding cloud service provider API or cloud serviceprovider portal. Instead, third-party orchestration modules 10404 managethe setup in response to service requests made by the serviceprovisioning engine 10014 via a common interface.

Third-party orchestration module 10404 may be configured withconnectivity information for communicating with respective cloud serviceproviders, including respective cloud service provider orchestrationsystems. Cloud service providers may update third-party orchestrationmodules 10404 by pushing up-to-date new or modified service catalogs andservice pricing information. The cloud exchange provider may thereforeavoid having to pull this information from the cloud service providers.

Each of third-party orchestration modules 10404 may represent anapplication executing on one or more data center servers of the cloudexchange and administered by the cloud exchange provider; a softwareplugin, module, or linked library; or another machine-executable codeexecutable in conjunction with the programmable network platform andcapable of satisfying service requests for third-party orchestration inthe manner described above.

By enabling consolidation of the setup and management of cloud servicesfrom third-party cloud service providers within the programmable networkplatform, third-party orchestration modules 10404 allow the cloudservice provider to become an authoritative owner of an end-to-endservice that include at least one component service (or “micro-service”)provided by a cloud service provider. In some cases, the end-to-endservice may include multiple micro-services from multiple differentcloud service providers each associated with a different third-partyorchestration module 10404. The end-to-end service may also include oneor more native services (such as any of the NFVs described herein)applied by the cloud exchange. Consolidation of service provisioningpermits the cloud exchange provider to offer unified billing tocustomers, whereby the cloud exchange provider bills the customer for acloud service provider-provided service according to cost informationreceived from the cloud service provider and passes payment through tothe cloud service provider. The cloud exchange provider also bills thecustomer for any native services, including layer 3 or otherconnectivity, NFVs, etc.

FIG. 6 illustrates service provisioning, described as follows. Servicepolicy manager 10400 may receive a service request to configure aparticular service. The request may specify one of third party services10404, such as third party service 10404A. The request may require theuse of one or more ports, thereby invoking port service 10402A. Therequest may further require an L3 service, thereby invoking L3 service10402B. Service provisioning engine 10014 may perform a set ofoperations 10406-10410 to configure the requested service.

Service provisioning engine 10014 may perform one or more operations10406 that procure service data for the requested service. To procurethe service data, service provisioning engine 10014 may query and/orstore data with business gateway 10412 and/or STAD 10016, as previouslydescribed in FIG. 5. Business gateway 10412 may include one or more APIsfor interfacing with financial logic 10018, as illustrated in FIG. 5.For instance, service provisioning engine 10014 may send billinginformation for the requested service to business gateway 10412, whichmay send the information to financial logic 10018. Financial logic 10018may associate the billing information for the service with a particularaccount. Service provisioning engine 10014 may also store information inSTAD 10016 that identifies the service and one or more properties of theservice. For instance, if the service includes a particular geographiclocation, particular service level request, etc., such details alongwith an identifier of the service may be stored in STAD 10016. After theservice has been implemented and is being used, STAD 10016 metrics forthe service may be updated and stored in STAD 10016.

Upon procuring the service data to implement the requested service,service provisioning engine 10014 may perform one or more operations10408 to select one or more NFUs. Service provisioning engine 10014 mayselect the one or more NFUs based on the initial request or instructionsreceived from service selector 10012. For instance, if the requestspecifies a particular geographic location, service provisioning engine10014 may select an NFU for the particular geographical location. If therequest specifies a particular quantity or type of resources, serviceprovisioning engine 10014 may determine one or more NFUs that manage oneor more cloud exchange points with sufficient resources to satisfy theparticular quantity and/or type of resources requested. Serviceprovisioning engine 10014 may query STAD 10016 to determine quantitiesand/or type of resources managed by different NFUs.

Once service provisioning engine 10014 has selected an NFU to provisionthe requested service, service provisioning engine 10014 may perform oneor more operations to generate network provisioning data at 10410. Forinstance, service provisioning engine 10014 may translate a higher-levelservice request received from a customer into network provisioning datacomprising a more specific set of instructions and/or requests thatservice provisioning engine 10014 sends to one or more NFUs. An NFU thatreceives the network provisioning data may configure networkinfrastructure based on the network provisioning data to provide therequested service.

FIG. 7 is a block diagram illustrating service assurance engine 10030 ofFIG. 5 in further detail, in accordance with one or more techniques ofthe present disclosure. As described in FIG. 5, service assurance engine10030 may receive information from network assurance engine 10028 andmay compare the information with service level information, such asservice level agreements, included in STAD 10016. By comparinginformation about the performance of infrastructure with service levelinformation in STAD 10016, service assurance engine may send servicelevel information to customers using one or more service assurance APIs,and whether such service level agreements or other performancethresholds are being met.

To determine whether such service level agreements or other performancethresholds are being met, service assurance engine 10030 may perform oneor more operations 16000-10608. Service assurance engine 10030 mayretrieve service data 1600. Service data may include definitions and/ordescriptions of services that have been requested by customers andprovisioned at one or more cloud exchange points. Based on the servicedata, service assurance engine 10030 may monitor the actual performanceof cloud exchange points that provide the requested services. Todetermine the actual performance of cloud exchange points, serviceassurance engine 10030 may query or otherwise receive performance data10602 from STAD 10030 that is populated by CNC system 10002 with datafrom one or more components, such as NFU 10004.

Service assurance engine 10030 may analyze the performance data inconjunction with the service data to identify anomalies, problems, ordeficient performance associated with a service provisioned by a cloudexchange point 10604. For example, service assurance engine 10030 maydetermine that one or more conditions, criteria and/or thresholds aresatisfied that indicate an anomaly, problem, or deficient performance10604. If such conditions, criteria and/or thresholds are satisfied,service assurance engine 10030 may perform one or more remedial actions.

Service assurance engine 10030 may store one or more remedial actions10606. In some examples, remedial actions may refer to one or moreoperations that may be taken by one or more components of CNC system10002 to remedy an anomaly, problem, or deficient performance. Aremedial action may be associated at service assurance engine 10030 withone or more conditions. For instance, when service assurance engine10030 determines that one or more conditions, criteria and/or thresholdsare satisfied, service assurance engine 10030 may determine remedialactions associated with the one or more conditions, criteria and/orthresholds. Examples of remedial actions may include re-allocatingresources to continue providing a particular service at a cloud exchangepoint, and sending one or more notifications to one or more recipients,to name only a few examples. In some instances, an administrator and/orcustomer may configure the remedial actions and/or one or moreconditions, criteria and/or thresholds prior to or at the time a serviceis provisioned at a cloud exchange point. In this way, if the one ormore conditions, criteria and/or thresholds are satisfied with respectto a cloud exchange point, service assurance engine 10030 may determinethe corresponding remedial actions.

As shown in FIG. 7, if the one or more conditions, criteria and/orthresholds are satisfied with respect to a cloud exchange point, serviceassurance engine 10030 may execute one or more remedial actions 10608.To execute the remedial actions, service assurance engine 10030 maycommunicate the remedial actions to service provisioning engine 10014,which carries out the operations defined by the remedial actions. Inthis way, service assurance engine 10030 may monitor and respond, in anautomated manner, to an anomaly, problem, or deficient performance byperforming one or more remedial actions.

FIG. 8 is a block diagram illustrating network provisioning engine 10024of FIG. 5 in further detail, in accordance with one or more techniquesof the present disclosure. Network provisioning engine 10024 of mayreceive network service definitions, instructions, and/or requests fromservice provisioning engine 10014 of CNC system 10002. Networkprovisioning engine 10024 uses the network service definitions,instructions, and/or requests to configure infrastructure managed by NFU10004 in order to provision network services at one or more cloudexchange points. A “network service definition,” as used herein, is datadefining parameters for provisioning a network service at leastpartially instantiable by configuring a network of network devices thatoffer network services. Network services may include network servicesprovided by (native service) or delivered by (cloud service orthird-party service) the cloud exchange to a consumer of theaforementioned network service(s).

To further illustrate, in FIG. 8, upon receiving a network servicedefinition, one or more instructions and/or requests, networkprovisioning engine 10024 may verify the contents and format of the anetwork service definition, instructions and/or requests 10800. Forexample, network provisioning engine 10024 may determine whether thenetwork service definition, instructions and/or requests are valid. Ifthe contents and/or format are invalid, network provisioning engine10024 may send a response to service provisioning engine 10014indicating the invalidity of the contents and/or format.

If the contents and format of the a network service definition,instructions and/or requests are valid, network provisioning engine10802 may choose a vendor to provide the service based on the a networkservice definition, instructions and/or requests. For instance, CNCsystem 10002 may allow a customer or cloud service provider to selectfrom a set of vendor equipment, one or more particular types of vendorequipment to provide a particular service. As an example, a cloudservice provider may specify a particular vendor to provide a firewallservice. The a network service definition, instructions and/or requestsreceived by network provisioning engine 10024 from CNC system 10002 mayspecify a particular vendor to provide the service. Network provisioningengine 10024 may determine whether equipment for the particular vendoris available to provide the service. If not available, networkprovisioning engine 10024 may send a response to service provisioningengine 10014 indicating the unavailability of the vendor equipment.

If equipment for the particular vendor is available, networkprovisioning engine 10024 may choose a particular form factor for thevendor equipment 10804. In some examples, the form factor may bespecified based on the a network service definition, instructions and/orrequests received by network provisioning engine 10024 from serviceprovisioning engine 10014. In other examples, network provisioningengine 10024 may automatically determine the form factor for the vendorequipment based on one or more parameters in the a network servicedefinition, instructions and/or requests. For example, the one or moreparameters may not specify the form factor; however, networkprovisioning engine 10024 may determine, based on the parameters, that aparticular form factor of vendor equipment will satisfy the requirementsof the a network service definition, instructions and/or requests. Forinstance, the parameters may specify a particular type of functionalityand/or resource requirement that network provisioning engine 10024 mayuse to determine which form factor of vendor equipment can satisfy therequirements.

Network provisioning engine 10024 may determine sizing for the vendorequipment and/or verify capacity of the vendor equipment 10806. In someexamples, network provisioning engine 10024 may query NTAD 10026 todetermine current resource allocation and usage for infrastructure of acloud exchange point that is managed by network provisioning engine10024. The infrastructure may include the vendor equipment identified bynetwork provisioning engine 10024 for the a network service definition,instructions and/or requests received from service provisioning engine10014. Based on the requirements of the instructions and/or requestsreceived from CNC 10002 and the current resource allocation and usagefrom NTAD 10026, network provisioning engine 10024 may determine whetheradequate resources exist at the vendor equipment determined by networkprovisioning engine 10024 to provision the requested service. If sizingand/or capacity for the vendor equipment is insufficient for theinstructions and/or requests received from service provisioning engine10014, network provisioning engine 10024 may send a response to serviceprovisioning engine 10014 indicating the insufficient sizing and/orcapacity for the vendor equipment.

If sizing and/or capacity for the vendor equipment is sufficient for thea network service definition, instructions and/or requests received fromservice provisioning engine 10014, network provisioning engine 10024 mayobtain a device handler to the specific vendor equipment 10808. In someexamples, a device handler may be an identifier that uniquely identifiesa particular device. Such devices may include the vendor equipmentincluded in the infrastructure of one or more cloud exchange points.Upon choosing the device handler, network provisioning engine 10024 mayconfigure or otherwise send one or more requests and/or instructions toSDN controller 10006 and/or hardware configurators 10008 to configurethe device identified by the device handler 10810. In some examples, therequests may specify create, read, update, or delete operations toperform with respect to the device identified by the device handler.

FIG. 9 is a block diagram illustrating network assurance engine 10028 ofFIG. 5 in further detail, in accordance with one or more techniques ofthe present disclosure. As described in FIG. 5, network assurance engine10028 may send information to service assurance engine 10030, which maycompare the information with service level information, such as servicelevel agreements, included in STAD 10016. Network assurance engine 10030may query NTAD 10026 in a similarly manner that service assurance engine10030 queries STAD 10016. By comparing information about the performanceof infrastructure with service level information in NTAD 10026, networkassurance engine may send network level information to customers usingone or more network assurance APIs, and whether service level agreementsor other performance thresholds are being met.

To determine whether such service level agreements or other performancethresholds are being met, network assurance engine 10028 may perform oneor more operations 11000-110010. Network assurance engine 10028 mayretrieve network data 11000. Network data may include actual bandwidth,dropped packets, latency, uptime, to name only a few examples. Based onthe network data, network assurance engine 10028 may monitor the actualperformance of cloud exchange points that provide the requestedservices. To determine the actual performance of cloud exchange points,network assurance engine 10028 may query or otherwise receiveperformance data 11002 from NTAD 10026 that is populated by NFU 10004.

Network assurance engine 10028 may analyze the performance data inconjunction with the service data to identify anomalies, problems, ordeficient performance associated with network infrastructure of a cloudexchange point 11004. For example, network assurance engine 10028 maydetermine that one or more conditions, criteria and/or thresholds aresatisfied that indicate an anomaly, problem, or deficient performance11004. If such conditions, criteria and/or thresholds are satisfied,network assurance engine 11004 may perform one or more remedial actions.

Network assurance engine 10028 may store one or more remedial actions11006. In some examples, remedial actions may refer to one or moreoperations that may be taken by one or more components of CNC system10002 to remedy an anomaly, problem, or deficient performance. Aremedial action may be associated at network assurance engine 10028 withone or more conditions. For instance, when network assurance engine10028 determines that one or more conditions, criteria and/or thresholdsare satisfied, network assurance engine 10028 may determine remedialactions associated with the one or more conditions, criteria and/orthresholds. Examples of remedial actions may indicate re-allocatingresources to continue providing a particular service at a cloud exchangepoint and/or sending one or more notifications to one or morerecipients, to name only a few examples. In some instances, anadministrator and/or customer may configure the remedial actions and/orone or more conditions, criteria and/or thresholds prior to or at thetime a service is provisioned at a cloud exchange point. In this way, ifthe one or more conditions, criteria and/or thresholds are satisfiedwith respect to a cloud exchange point, network assurance engine 10028may determine the corresponding remedial actions.

As shown in FIG. 9, if the one or more conditions, criteria and/orthresholds are satisfied with respect to a cloud exchange point, networkassurance engine 10028 may execute one or more remedial actions 11008.To execute the remedial actions, network assurance engine 10028 maycommunicate the remedial actions to network provisioning engine 10024,which carries out the operations defined by the remedial actions. Inthis way, network assurance engine 10028 may monitor and respond, in anautomated manner, to an anomaly, problem, or deficient performance byperforming one or more remedial actions.

FIG. 10 is a block diagram illustrating a programmable network platform11600, in accordance with one or more techniques of the presentdisclosure. Programmable network platform 11600 may represent an exampleinstance of programmable network platform 120, programmable networkplatform 10000, or other programmable network platform described in thisdisclosure. As shown in FIG. 10, programmable network platform 11600 mayinclude a centralized network control (CNC) system 11601 that controlsdata fabric 11614. Data fabric 11614 may be configured by CNC system11601 to provide one or more services, including virtual connections,which allow customers 11610 to use services provided by cloud serviceproviders 11616.

Customers 11610 may desire to directly cross-connect to cloud serviceproviders 11616 at a common point, such as data fabric 11614, therebyallowing direct exchange of network traffic between the networks of thecustomers and cloud service providers. In some examples, one or moreservices may be applied by data fabric 11614 to network trafficforwarded between customers 11610 and cloud service providers 11616. Forinstance, a customer may configure an L3 connection service with afirewall between the customer and a cloud service provider using datafabric 11614.

As further illustrated in FIG. 16, infrastructure that implements datafabric 11614 may be logically divided as edge and core networkinfrastructure. Edge network infrastructure may include network devicesthat couple the core network of data fabric 11614 to customer and cloudservice provider networks. Core network infrastructure may includenetwork devices that forward network traffic through the core network ofdata fabric 11614. In the example of FIG. 10, CNC system 11601 includesedge network control module 11618 that configures and provisionsservices at network devices included in the edge network infrastructureof data fabric 11614. CNC system 16601 also includes core networkcontrol module 11620 that configures and provisions services at networkdevices included in the core network infrastructure of data fabric11614. Although shown in CNC system 11601, functionality of edge and/orcore network control modules 11618, 11620 may be included in one or moreNFUs and/or distributed between NFUs and CNC system 11601.

In one example, a customer of customers 11610 may desire to access oneor more services provided by cloud service providers 11616. As anexample, the customer may desire to access an office productivityservice of cloud service providers 11616. Accordingly, the customer maysubmit a request using user portal 11602 for an L3 connection andfirewall provided at data fabric 11614 that allows for the directexchange of network traffic between the customer and the cloud serviceprovider of the office productivity service. Example user interfacesprovided by user portal 11602 are illustrated in FIGS. 11 and 12.

CNC system 11601 receives the request for the L3 connection service.Edge network control module 11618 and core network control module 11620may each configure edge network infrastructure and core networkinfrastructure, respectively, for the L3 connection service. In someexamples, edge and core network control modules 11618, 11620 mayidentify one or more NFUs that will configure network infrastructure toprovide the L3 connection service. For instance, edge network control11618 may identify a first set of one or more NFUs to configure edgenetwork infrastructure, while core network control module 11620 mayidentify a second set of one or more NFUs to configure core networkinfrastructure.

In some examples, network control 11618 may directly configure edgenetwork infrastructure that couples the customer network to the corenetwork of data fabric 11614 and the cloud service provider 11616 to thecore network of data fabric 11614. In other examples, edge networkcontrol module 11618 may send instructions and/or requests for the L3connection service and firewall service to one or more NFUs thatconfigure the edge network infrastructure that couples the customernetwork to the core network of data fabric 11614 and the cloud serviceprovider 11616 to the core network of data fabric 11614. Similarly, corenetwork control module 11620 may send instructions and/or requests forthe L3 connection service and firewall to one or more NFUs thatconfigure the core network infrastructure of data fabric 11614 and thecloud service provider 11616 to the core network of data fabric 11614.Upon configuration of edge and core network infrastructure to providethe L3 connection service and firewall, the customer of customers 11610that requested the L3 connection service may directly connect, via anetwork service provider and data fabric 11614, to the officeproductivity service.

In some examples, one or more IT systems 11604 may be coupled to CNCsystem 11601. IT systems 11604 may include one or more computingdevices, such as desktop computers, tablets, smartphones, and servers,to name only a few examples. IT systems 11604 may provide one or moreuser interfaces to administrators, which may use IT systems 11604 toadministrate 11601. IT systems 11604 may, for example, receive userinputs to configure CNC system 11601. Based on the user inputs, ITsystems 11604 may send requests and/or instructions to CNC system 11601.In some examples, CNC system 11601 may provide or otherwise expose oneor more RESTful interfaces that can be invoked by IT systems 11604.

FIG. 11 is a block diagram illustrating an example user interface 12200to request a service, in accordance with one or more techniques of thepresent disclosure. In some examples, a centralized network controlsystem, such as CNC system 10002 or CNC system 11601, or a portal tosuch as system, may generate user interface 12200 for display. In someexamples, user interface 12200 may be implemented as one or more HTMLdocuments that may be rendered in a web browser. User interface 12200may be implemented in a standalone application that is executable on amobile computing device, desktop computing device, or laptop device toname only a few examples, and that invokes a programmable networkplatform in a manner described herein. For example purposes, userinterface 12200 is illustrated in a web browser in FIG. 11.

As shown in FIG. 11, user interface 12200 may allow a user to configurean L3 connection service. User interface 12200 may include a side menu12202, which lists each different type of service that may be configuredby a user. Side menu 12202 may include one or more elements, where eachrespective element corresponds to a particular type of service. Anelement may be selected by a user, which displays a corresponding userinterface to configure the type of service associated with the element.For instance, as shown in FIG. 11, an element that corresponds toconfiguring a L3 connection service has been selected by the user.Accordingly, user interface 12200 includes user interface elements toconfigure a L3 connection service.

In some examples, user interface 12200 includes a user interface element12004, such as a label, that displays the type of service definitionbeing configured by the user. In the example of FIG. 11, user interfaceelement 12004 includes “L3 Connect” as the type of service definitionthat is being configured by the user, which may correspond to a L3connection service. As used herein, the term “service definition” refersto data defining parameters for provisioning a business level service,within a cloud exchange, for one or more network services provided by(native service) or delivered by (cloud service or third-party service)the cloud exchange to a consumer of the aforementioned service(s). Aservice definition may define multiple services within an overallservice, including for each service one or more service requirements toimplement the service. Implementing a service may include both serviceorchestration and network provisioning, for instance. Native servicesmay include, e.g., a port service, L3 connectivity service, L2 service,L3+ service, firewall, NAT, DPI, and other native services appliedwithin the cloud exchange to cloud service traffic from a cloud serviceprovider network to modify, inspect, shape (e.g., filter or apply QoS),and/or deliver the cloud service traffic. Cloud services may includeSoftware-as-a-Service (SaaS), Platform-aaS (PaaS), Infrastructure-aaS(IaaS), Virtualization-aaS (VaaS), and data Storage-aaS (dSaaS)services, such as content/media delivery, cloud-based storage, cloudcomputing, online gaming, IT services, etc. The service definition mayspecify cloud exchange endpoints and other connectivity information forconnecting to a cloud service, policies, SLA, and/or QoS for a service,an originator, an owner, a service identifier, a destination, forexample. Additional examples of service definitions are described below.Certain parameters of the service definition, such as bandwidth,policies, SLA, and QoS for the service to be applied within the cloudexchange, may be alternatively referred to as “service requirements” inthat the requestor requires that the service be applied/delivered insuch as manner as to meet such requirements.

In some examples, user interface 12200 may include a user interfaceelement 12006 that enables a user to select or otherwise specify ageographic location. The geographic location may be a location of acustomer site and/or a location of a cloud exchange point. For instance,in FIG. 11 the user may use user interface element 12006 to provide ageographic location in San Jose, Calif. To name only a few examples,user interface element 12006 may be implemented as a drop-down menu thatis pre-populated with available locations or may be implemented as atext input field in which the user may enter a location.

User interface 12200 may include a user interface element 12008 thatenables a user to select or otherwise specify a network segment for theL3 connection service for the customer. In some examples, the networksegment is defined by a range of layer 3 addresses, such as InternetProtocol (IP) addresses 12008. For instance, in FIG. 11 the user may useuser interface element 12008 to provide an IP address range of10.10.10.0/24. To name only a few examples, user interface element 12008may be implemented as a drop-down menu that is pre-populated withavailable network segment values or may be implemented as a text inputfield in which the user may enter a network segment value.

User interface 12200 may include one or more user interface elements12010 that enable a user to select or otherwise specify minimum and/ormaximum performance requirements for the L3 connection service. Forinstance, a variety of bandwidths may be associated with the one or moreuser interface elements 12010 from which the user may select a minimumand/or maximum performance requirement for the service. In the exampleof FIG. 11, example performance requirements, selectable by the user,include 1 Mbps, 10 Mbps, 100 Mbps, 1 Gbps, 10 Gbps, 40 Gbps, and 100Gbps. To name only a few examples, user interface element 12010 may beimplemented as a drop-down menu that is pre-populated with availablebandwidths, a set of radio buttons or checklists where each selectableelement has a corresponding bandwidth, or may be implemented as a textinput field in which the user may enter a bandwidth.

User interface 12200 may include one or more user interface elements12012, 12014, and 12016 that enable a user to, respectively, specifywhether the bandwidth is burstable, specify an Excess Information Rate(EIR), and/or specify a maximum latency. As shown in FIG. 11, userinterface element 12012 may be implemented as a checkbox, user interfaceelement 12012 may be implemented as a text input field, and userinterface element 12016 may be implemented as a text input field,although other types of user interface elements may be also be used.

User interface 12200 may include one or more user interface elements12018 that enable a user to select or otherwise specify an uptime oravailability requirement for the L3 connection service. For instance, avariety of availability levels may be associated with the one or moreuser interface elements 12018 from which the user may select an uptimeor availability requirement for the service. In the example of FIG. 11,example uptime or availability requirements, selectable by the user,include 99.9999%, 99.999%, 99.995%, 99.99%, 99%, and 90% uptime. To nameonly a few examples, user interface element 12018 may be implemented asa drop-down menu that is pre-populated with uptime or availabilityvalues, a set of radio buttons or checklists where each selectableelement has a corresponding uptime or availability, or may beimplemented as a text input field in which the user may enter an uptimeor availability.

User interface 12200 may include one or more user interface elements12020 that enable a user to select or otherwise specify a cloud serviceprovider for the L3 connection service. For instance, the customer mayselect one or more cloud service providers that provide services whichthe customer may consume using the L3 connection service. To name only afew examples, user interface elements 12020 may be implemented as adrop-down menu that is pre-populated with cloud service providers, a setof radio buttons or checklists where each selectable element has acorresponding cloud service provider, or may be implemented as a textinput field in which the user may enter a cloud service provider. Userinterface 12200 may include a user interface element 12022 that enablesa user to select or otherwise specify a network segment for the L3connection service for the cloud service provider. In some examples, thenetwork segment is defined by a range of layer 3 addresses, such asInternet Protocol (IP) addresses 12022.

User interface 12200 may include one or more user interface elements12024, 12026, and 12028 that enable a user to, respectively, selectDistributed Denial-of-Service protection for the service, theDistributed Denial-of-Service protection provider, and/or a redirect IPaddress. As shown in FIG. 11, user interface element 12024 may beimplemented as a radio button or checkbox, user interface element 12026may be implemented as a dropdown list, and user interface element 12028may be implemented as a text input field, although other types of userinterface elements may be also be used.

User interface 12200 may include a user interface element 12030 tosubmit the selected and/or inputted values of user interface 12200 forfurther processing. User interface element 12030 is implemented as abutton in FIG. 11. When user interface element 12030 is selected, theselected and/or inputted values of user interface 12200 may be validatedand may be further processed to implement the service and/or provide anestimate of cost to implement the service to the user before actuallyimplementing the service.

FIG. 12 is a block diagram illustrating an example user interface 12400to display a cost estimate for a service, in accordance with one or moretechniques of the present disclosure. In some examples, a centralizednetwork control system, such as CNC system 10002 or CNC system 11601, ora portal to such as system, may generate user interface 12400 fordisplay. In some examples, user interface 12400 may be implemented asone or more HTML documents that may be rendered in a web browser. Userinterface 12400 may be implemented in a standalone application that isexecutable on a mobile computing device, desktop computing device, orlaptop device to name only a few examples, and that invokes aprogrammable network platform in a manner described herein. For examplepurposes, user interface 12400 is illustrated in a web browser in FIG.12.

As shown in FIG. 12, user interface 12400 may provide cost informationfor a service based on user input provided for user interface 12200 inFIG. 11. In some examples, user interface 12400 may include one or moreuser interface elements 12402 that include the input values provided bythe user in user interface 12200 of FIG. 11. Accordingly, a user mayreview the input values to determine that the value are correct. If theuser desires to change one or more values, the user may select userinterface element 12204, which causes user interface 12200 to again bedisplayed with the input values, thereby allowing the user to update orotherwise change such input values. In other examples, user interfaceelement 12204 may include user interface elements that allow the user toupdate or otherwise make changes to input values in user interface 12204without returning to user interface 12200.

User interface 12400 may include one or more user interface elementsthat included one or more costs for the requested service. For instance,user interface 12400 may include user interface element 12406, whichdisplays a cost-per-month of the requested service. By outputting thecost-per-month for the requested service, the user may evaluate the costof the service. In some examples, other cost values may be included inuser interface 12400. For instance, user interface 12400 may includeitemized costs corresponding to one or more input values of userinterface elements 12402. In some examples, alternative itemized costsfor alternative input values may also be included in user interface12400. For instance, the alternative cost for 99.999% uptime may beincluded in user interface 12400. The user, upon reviewing the costinformation, may select user interface element 12406 (e.g., a button) tosubmit the request for the service. Upon selecting the user interfaceelement, CNC system 10002 may configure one or more cloud exchangepoints to provision the service requested by the user based on the inputvalues shown in user interface 12400.

FIG. 13 is a conceptual diagram illustrating example components for aprogrammable network platform operating according to techniquesdescribed in this disclosure. In this example, programmable networkplatform 12500 includes a centralized network control component 12504(“CNC 12504”) that interfaces with a plurality of decentralized networkfield units 12508A-12508C (“NFUs 12508”) to provision devices of edgenetwork 12600 and assure the delivery of layer 3 cloud services tocustomers. Each of the NFUs 12508 may provision a different subset ofdevices, or “portion of,” edge network 12600. For example, edge network12600 may include devices distributed among numerous cloud exchangepoints or metro-based cloud exchanges, each cloud exchange point ormetro-based cloud exchange being provisioned for services by a differentNFU 12508.

Programmable network platform 12500 may represent, e.g., an exampleinstance of programmable network platform 120. CNC 12504 may representan example instance of CNC 1002 or CNC 11601. NFUs 12508 may eachrepresent an example instance of NFU 1004. User portal 12502 representsclient-side software for interfacing with the programmable networkplatform 12500 and may represent a customer portal, customerapplications, a cloud exchange provider application, a console such as acommand-line interface or graphical user interface, and/or a cloudservice provider-developed application. Users/clients may includecustomers, the cloud exchange provider, and cloud service providers.

In some aspects, a controller, such as the programmable network platformdescribed herein, may provision the cloud exchange with services made upof multiple constituent services provided by different cloud serviceproviders. Each of these constituent services is referred to herein as a“micro-service” in that it is part of an overall service applied toservice traffic. That is, a plurality of micro-services may be appliedto service traffic in a particular “arrangement,” “ordering,” or“topology,” in order to make up an overall service for the servicetraffic. Micro-services may be applied by cloud service providers orwithin the cloud exchange.

The programmable network platform may in this way orchestrate abusiness-level service across heterogeneous service providers. Theprogrammable network platform exposes interfaces by which a portal,console (e.g., user interface application), or other application maydefine the service policy, quality, service level agreements (SLAs), andcost as a coordinated service topology made up of micro-servicesprovided by different cloud service providers (or “cloud vendors”). Eachmicro-service may have a corresponding service policy, quality, SLA, andcost, as part of the overall, end-to-end business service definition, asdescribed in further detail below. When provided with a servicedefinition for an end-to-end service having multiple componentmicro-services, the programmable network platform may orchestrate eachof the micro-services within the cloud exchange and stitch themicro-services together according to the defined topology in order toreify the end-to-end service within the cloud exchange (or edge networkthat includes the cloud exchange). As a result, the cloud exchangeinterconnects, in the data plane, micro-services provided by respectivecloud services providers on behalf of and for the benefit of a customerof the cloud exchange or of at least one of the cloud service providers.

FIG. 14A is a block diagram that illustrates an example configuration ofa programmable edge network that has been configured to apply multiplenative services to cloud service traffic aggregated by a cloud exchangefrom multiple cloud service providers for delivery to a customer. Edgenetwork 12600 may include any of the data center-based cloud exchangesor cloud exchange points described herein, such as cloud exchange points128 of FIG. 1, cloud exchange 200 of FIG. 2, and the cloud exchangepoint of FIG. 10 including data fabric 11614.

Edge network 12600 includes network infrastructure including layer 3(L3) forwarding elements 12622A-12622B (collectively, “forwardingelements 12622”), which may include one or more routers, switches, andother L3 forwarding devices. Although not shown, edge network 12600 mayalso include, for example, one or more non-edge (core) switches,routers, hubs, gateways, security devices such as firewalls, intrusiondetection, and/or intrusion prevention devices, computer terminals,laptops, printers, databases, wireless mobile devices such as cellularphones or personal digital assistants, wireless access points, bridges,cable modems, application accelerators, or other network devices.

Edge network 12600 further includes servers 12640A-12640B (collectively,“servers 12640”) that offer one or more compute/computing farms by whichthe edge network 12600 may offer services to customer 12604 and/or applyservices to service traffic for customer 12604. Servers 12640 mayrepresent x86 or other real or general-purpose servers configured toapply and/or offer services to customers. Servers 12640 may also includespecial-purpose appliances or containers for applying services toservice traffic between customers and cloud service providers 12606.Such services may include, e.g., NAT, DPI, FW, DDOS mitigation, andother native services that may be applied by the cloud exchange edgenetwork 12600 controlled by the cloud exchange provider and asconfigured by programmable network platform 12500.

The cloud exchange provider that manages, administers, and configuresedge network 12600 facilitates the application of such native servicesto service traffic exchanged between customer 12604 and any of cloudservice providers 12606A-12606D (“CSPs 12606”), each of which mayrepresent any of cloud service providers 110 for instance. Edge network12600 is configured with virtual NAT (vNAT) service 12614 forapplication to service traffic sourced by or destined to CSP 12606A,virtual Deep Packet Inspection (vDPI) service 12616 for application toservice traffic sourced by or destined to CSP 12606B, and virtualFirewall (vFW) service 12618 for application to service traffic sourcedby or destined to CSP 12606C, all such service traffic sourced by ordestined to customer 12604. Services 12614, 12616, and 12618 mayrepresent Network Function Virtualization (NFV) services in that theservices virtualize functions frequently offered by network serviceproviders employing dedicated service appliances (e.g., NAT, DPI, andfirewall devices, whether employed separately or by an integratedUnified Threat Management (UMT) device, e.g.). While this exampleillustrates and describes virtual services (or NFVs), the services maybe applied by controllers, appliances, or containers administered by thecloud exchange provider.

PE router 12602 in this example represents a real or virtual PE routerthat aggregates service traffic from multiple cloud service providers12606 for delivery to a single customer 12604. Programmable networkplatform 12500 configures the PE router 12602 to import and export L3routes for the cloud service providers 12606 to enable aggregated layer3 cloud service cloud service delivery, as described above with respectto FIGS. 1-4. In addition, programmable network platform 12500 allowscustomers, cloud service providers, and/or the cloud exchange providerto configure edge network 12600 with services 12614, 12616, and 12618for assured delivery of cloud service traffic from respective cloudservice providers 12606A-12606C.

In some aspects, a controller, such as the programmable network platformdescribed herein, may provision a L3 cloud-based services exchange(“cloud exchange”) to deliver services made up of multiple constituentservices provided by different cloud service providers and in some casesby the cloud exchange itself. Each of these constituent services isreferred to herein as a “micro-service” in that it is part of an overallservice applied to service traffic. That is, a plurality ofmicro-services may be applied to service traffic in a particular“arrangement,” “ordering,” or “topology,” in order to make up an overallservice for the service traffic. The micro-services themselves may beapplied or offered by the cloud service providers.

The programmable network platform may in this way orchestrate abusiness-level service across heterogeneous cloud service providers. Theprogrammable network platform exposes application programming interfaces(APIs) by which a portal, console (e.g., user interface application), orother application may define the service policy, quality, service levelagreements (SLAs), and cost as a coordinated service topology made up ofmicro-services provided by different cloud service providers (or “cloudvendors”). Each micro-service may have a corresponding service policy,quality, SLA, and cost, as part of the overall, end-to-end businessservice definition, as described in further detail below. When providedwith a service definition for an end-to-end service having multiplecomponent micro-services, the programmable network platform orchestrateseach of the micro-services within the cloud exchange and stitches themicro-services together according to the defined topology in order toreify the end-to-end service within the cloud exchange data plane (e.g.,an edge network for the cloud exchange). As a result, the cloud exchangeinterconnects, in the data plane, micro-services provided by respectivecloud services providers on behalf of and for the benefit of a customerof the cloud exchange. In doing so, the cloud exchange provider mayfacilitate business transactions between the cloud service providers andcustomers.

FIG. 14B is a block diagram that illustrates an example configuration ofa programmable edge network that has been configured to offer anend-to-end service that is a sequence of multiple constituentmicro-services applied by respective cloud service providers. Edgenetwork 12600 may include any of the data center-based cloud exchangesor cloud exchange points described herein, such as cloud exchange points128 of FIG. 1, cloud exchange 200 of FIG. 2, and the cloud exchangepoint of FIG. 10 including data fabric 11614.

Micro-services for an overall service established for a customer mayinclude a mix of Software-as-a-Service (SaaS), Platform-aaS (PaaS),Infrastructure-aaS (IaaS), Virtualization-aaS (VaaS), and dataStorage-aaS (dSaaS) services in any ordering. For example, differentcloud service providers 12606 may execute applications that analyzeapplication data of service traffic 12612 to generate reporting data,store application data, generate new application data for sending asadditional service traffic 12612 along the sequence of micro-services,and so forth.

In accordance with techniques described herein, each of cloud serviceproviders 12606 offers/executes a micro-service that edge network 12600arranges (or “chains”) together to form an overall multi-cloud servicefor customer 12604. More specifically, in some aspects, the programmablenetwork platform 12500 configures a router (or forwarder) 12602 tostitch together the micro-services offered by respective various cloudservice providers 12606 into an overall service to apply to packets ofservice traffic 12612.

The customer 12604 may use the programmable network platform 12500 toselect and arrange the micro-services of cloud service providers 12606for at least some of the service traffic originated or received by thecustomer 12604 network. As described above, the programmable networkplatform 12500 may offer the customer connectivity to multiple differentcloud service providers. Upon selecting the cloud service providersoffering micro-services and a topology for the micro-services, theprogrammable network platform 12500 configures the edge network 12600 toprovision connectivity for the micro-services for the customer 12604.Selecting a cloud service provider may include entering connectivityparameters for the micro-service offered by the cloud service provider.Such connectivity parameters may include L3 routes and bandwidth orother QoS requirements.

In the illustrated example, router 12602 receives L3 routes for each ofthe cloud service provider 12606 networks that enable the router 12602to forward service traffic 12612 along the overall end-to-end servicepath. To implement the router 12602, the programmable network platform12500 may, for instance, configure one or more servers 12620A to executea virtual router (or configure a dedicated router) that includes VRFsfor each of the cloud service provider 12606 networks. As describedabove with respect to FIG. 4, for instance, the VRFs may be associatedwith route targets to establish a hub-and-spoke topology for sending andreceiving service traffic 12612, with router 12602, to and from thecloud service provider 12606 networks that offer the micro-services.

Consequently, the cloud exchange provider that administers edge network12600, using via the programmable network platform 12500, may alleviatecustomer 12604 from establishing, administering, and at least in someinstances assuring the end-to-end service that is made up ofmicro-services of cloud service providers 12606. Customer 12604, forinstance, can forward service traffic 12612 to edge network 12600 inaccordance with cloud exchange provider routes and need not peer withcloud service provider 12606 networks in order to obtain routes for eachof those networks. Rather, the cloud exchange point of edge network12600 internalizes the L3 routing protocol peering arrangements with thecloud service provider 12606 networks and imports the L3 routes to cloudservice provider 12606 networks in order to forward service trafficalong the topology of the overall service.

Router 12602 may include VRFs configured by the programmable networkplatform to import and export respective L3 routes for the servicesprovided by cloud service providers 12606. The router 12602 may receivethe routes from the programmable network platform in some instances, orreceive the routes via peering sessions with the provider edge (PE)routers of edge network 12600 that connect the cloud exchange to any ofthe cloud service provider 12606 networks.

The edge network 12600 may advertise, to customer 12604, L3 routes ofthe cloud exchange point autonomous system NATed with L3 routes of thecloud service provider 12606D network by the cloud exchange, L3 routesfor the vNAT service 12614, (in this example that includes a NATservice), or L3 routes of the cloud service provider 12606D network. Inthis way, the edge network 12600 may aggregate the delivery of multiple,multi-cloud L3 services to customer 12604.

FIG. 14B illustrates the delivery, by edge network 12600, of anend-to-end service made up of multiple micro-services to service traffic12612. The customer 12604 network sends service traffic 12612 to edgenetwork 12600 and destined for a network address within a prefixadvertised as an L3 route by the edge network 12600 to the customer12604 network. Service traffic 12612 may include one or more packetflows, each packet flow associated with one or more packets that includeapplication-layer data generated and/or consumed by an applicationexecuting within the customer 12604 network. Although illustrated inFIG. 14B as originating from the customer 12604 network and proceedingupstream toward the cloud service providers, the techniques aresimilarly applicable to downstream service traffic destined for thecustomer 12604 network, as well as to downstream service trafficoriginated from one of the cloud service providers 12606 networks anddestined for one of the cloud service provider 12606 networks. Forexample, a cloud service provider 12606D may inject application data viarouter 12602 to an application executed by the cloud service provider12606C network to analyze the application data, which sends the analyzedapplication data for processing to the cloud service provider 12606Bnetwork, which in turns send the application data for storage to adSaaS-providing cloud service provider 12606A network.

In the illustrated example, however, router 12602 receives servicetraffic 12612, determines the first micro-service for service traffic12612, and directs the service traffic 12612 to the cloud serviceprovider 12606A network. The cloud service provider 12606A networkapplies its micro-service returns the service traffic 12612 (which maybe modified from the service traffic originated by the customer 12604 inaccordance with the micro-service applied by cloud service provider12606A) to router 12602.

Router 12602 determines the next micro-service for service traffic 12612and forwards the service traffic 12612 to cloud service provider 12606B.The cloud service provider 12606B network applies its micro-service andreturns the service traffic 12612 (which may be modified in accordancewith the micro-service applied by cloud service provider 12606B) torouter 12602.

Router 12602 determines the next micro-service for service traffic 12612and forwards the service traffic 12612 to cloud service provider 12606C.The cloud service provider 12606C network applies its micro-service andreturns the service traffic 12612 (which may be modified in accordancewith the micro-service applied by cloud service provider 12606C) torouter 12602. Router 12602 determines the next micro-service for servicetraffic 12612 and forwards the service traffic 12612 to cloud serviceprovider 12606D.

Again, in some examples, CSPs 12606 may originate and edge network 12600may deliver service traffic downstream to customer 12604, with edgenetwork 12600 applying a set of micro-services to such downstreamservice traffic. For instance, the cloud service provider 12606D networkmay include or otherwise represent a content delivery network (CDN). ACDN may offer streaming video, streaming audio, streaming multimedia,gaming content, or other content delivery services to customers, and inthis case to customer 12604 via the cloud exchange.

As a result, the edge network 12600 including a cloud exchangeinterconnects, in the data plane, micro-services provided by respectivecloud services providers 12606 on behalf of and for the benefit of acustomer 12604 of the cloud exchange or of at least one of the cloudservice providers.

When provided with a service definition for an end-to-end service havingmultiple component micro-services, a programmable network platform asdescribed herein may orchestrate each of the micro-services within thecloud exchange and stitch the micro-services together according to thedefined topology in order to reify the end-to-end service within thecloud exchange (or edge network that includes the cloud exchange). Inaccordance with techniques of this disclosure, the service definitionfor an end-to-end service may enable a user of the programmable networkplatform to define not only the end-to-end service but also the servicetopology in such a ways as to ensure the correct sequencing of themicro-services service chain. The data encapsulated in the data modelfor the service definition may also include the authoritative serviceowner for business purposes (e.g., billing and SLA assurance). The“user” may refer to a customer, the cloud exchange provider, or a cloudservice provider that is the authoritative service owner.

By using a data model for a multi-cloud, multi-service servicedefinition as described herein, the programmable network platform (orother orchestration systems such as SDN controllers or orchestrators)may be enabled to recognize a service request as a request for a set ofmicro-services that make up the entire service. In some examples, theservice definition includes several sections that will enable theprogrammable network platform to provide the service of chaining severalservices, whether of native services provided by the cloud exchangeprovider or of cloud services provided one or multiple cloud serviceproviders. That is, the cloud exchange provider that administers theprogrammable network platform is able to provide a service chain that,when given respective definitions for multiple micro-services and atopology (or sequence) for the multiple micro-services, interconnectsthe micro-services according to the topology to facilitate an end-to-endservice. The data model thus provides data with which the programmablenetwork platform can effectively instantiate the requested chain ofservices and to also ensure that the services thus rendered are chainedin the correct topology. The data model may be divided by theprogrammable network platform into one or more service requests that thenative programmable network platform for the cloud exchange may issue toother service orchestration systems to complete. Other serviceorchestration systems may include, e.g., SDN controllers and/ororchestration systems for cloud service providers that facilitateNFV-instantiation and service traffic routing to/from NFV instances.

A service definition conforming to a multi-cloud, multi-service datamodel of the described techniques may specify an overall end-to-endservice associated with one or more of (1) an originator, (2) an owner,(3) a identifier, (4) a destination, and (5) a topology. The originatorrefers to the end-to-end service requestor, typically but notexclusively a customer of the cloud exchange. The owner refers to theauthoritative service owner that, e.g., handles and is responsible forbilling and charging to the originator/customer on behalf of the cloudservice providers. The identifier uniquely identifies the end-to-endservice within the cloud exchange. The destination refers to the cloudexchange where the requested service is instantiated. The topologydetermines the sequence of an array of micro-services included in theservice definition.

Each micro-service defined within a service definition may be an elementof an array of micro-services. A micro-service may be associated in thedata model with one or more of (1) descriptive information, (2) a firstor “customer” endpoint, (3) a second or “cloud service provider”endpoint, (4) policies to be applied by the cloud exchange for themicro-service, (5) Quality-of-Service (QoS) parameters for themicro-service, and (6) a time range for the micro-service.

The descriptive information may include a unique identifier for themicro-service within the cloud exchange. The first endpoint for the datamodel may specify a customer identifier to which the cloud exchange isto attach for service delivery, and a service key. A service key is thelicense key obtained by a customer for purposes of instantiating andactivating a requested service. In the event a service is temporarilyrequested by the cloud exchange itself, the cloud exchange may obtainthe service key from the cloud service provider and use the service keyto instantiate and activate the service. The second endpoint for thedata model may specify a cloud service provider identifier to which thecloud exchange is to attach for service delivery, and a service key.Each endpoint description for the first and second endpoint may alsoinclude endpoint specific data, such as a metro location identifier,port identifiers, data center identifiers, virtual circuits and virtualcircuit bandwidth, profiles, and configuration, and so forth.

The policies may identify the configuration and settings to be appliedon corresponding micro-services. For example, a policy may includefirewall rules for a firewall micro-service within the service chain.Another policy may include packet inspection rules for a DPImicro-service. Another policy may specify the QoS to be applied for aQoS micro-service. The time range is used to specify the duration forwhich the service metrics are to be reported when querying the status ofthe service. In some examples, the time range is used only during theREAD operation of the service.

The programmable network service data model may be used by a serviceinterface of any of the examples of programmable network platformsdescribed herein to allow external applications to define a topology ofmicro-services. The following MS_API definition is an example of aprogrammable network service data model, according to techniquesdescribed herein:

MS_API {   SRVC_Tag   { SRVC_Orig: Varchar SRVC_Owner: VarcharSRVC_accID: Varchar SRVC_Dest: Varchar SRVC_Topology:Varchar } SRVC_Num:Integer // Number of SRVC_API elements SRVC_Value  { SRVC_API { //Micro-service 1 } SRVC_API { // Micro-service 2 } ... SRVC_API { //Micro-service N } } // End micro-service definitions } // End servicedefinition

The MS_API end-to-end service definition data model includes SRVC_Tagand SRVC_Value containers. The SRVC_Tag container includes valuesassociated with the overall service definition. Specifically, SRVC_Origspecifies the originator, SRVC_Owner specifies the authoritative ownerfor the service, SRVC_accID specifies the account identifier belongingto the service originator that identifies the originator to the cloudexchange provider, SRVC_Dest specifies the the cloud exchange where therequested service is instantiated, and SRVC_Topology specifies asequence of an array of micro-services specified in the SRVC_Valuecontainer.

The SRVC_Value is a micro-service definition container and includes oneor more micro-service definitions for respective micro-services. Thefollowing SRVC_API data model is an example of a micro-servicedefinition, e.g., for the MS_API definition model above:

SRVC_API { Srvc_Defn { Srvc_AccID: VarChar Srvc_Type: VarCharSrvc_Oper_Type: VarChar Srvc_ID: VarChar Target_SP: VarChar Srvc_Vendor:VarChar EndPoint1 { SP_ID: VarChar Srvc_Key: VarChar[ ] EP_Data { <DataSpecific for Defined Service>  } } // End Endpoint1 Definition EndPoint2{ SP_ID: VarChar Srvc_Key: VarChar[ ] EP_Data { <Data Specific forDefined Service> } } // End Endpoint2 Definition } // End ServiceDefinition Policy { <Data Specific to Policy Definition of DefinedService> } // End Policy Definition QOS {   <<Parameter:Configured_Value, Real_Value>> Availability: VarChar, VarChar Bandwidth:Num, Num Burstable: Boolean Burst_EIR: VarChar,V arCharClass_Of_Service: VarChar, VarChar { Latency: [VarChar,VarChar],[VarChar, VarChar] Jitter: [VarChar, VarChar, VarChar, VarChar]ErrorRate: [VarChar,VarChar],[VarChar, VarChar] PacketDrops:[VarChar,VarChar],[VarChar, VarChar]  } } // End QOS definitionTimeRange { Start_Time: VarChar End_Time: VarChar } // End Time RangeDefinition } // End SRVC_API

The above service definition includes service definition (“Srvc_Defn”),policy (“Policy”), quality of service (“QOS”), and time range(“TimeRange”) containers for specifying and obtaining characteristics ofa micro-service via the programmable network platform. The servicedefinition container specifies descriptive information for themicro-service, including Srvc_AccID, Srvc_Type, Srvc_Oper_Type, Srvc_ID,Target_SP, Srvc_Vendor. Srvc_AccID is a unique account identifier thatidentifies a particular service originator uniquely with the particulartarget service provider, Target_SP. Srvc_Type may be applicable onlywhen native cloud exchange services are provided and,defines the type ofcloud exchange service being delivered. Srvc_Oper_Type specifies theoperation type, i.e., one of the CRUD operations. Srvc_ID is to beprovided by the originator for an already existing (e.g., alreadycreated) service for which a Read, Update or Delete operations are to beperformed. Target_SP is the target service provider that provides therequested service

The EndPoint1 container defines a first endpoint of the cloud exchangefor the micro-service and specifies a service provider identifier(SP_ID) to which the cloud exchange is to attach for service deliveryusing a service key or license key. The Endpoint2 container defines asecond endpoint of the cloud exchange for the micro-service andspecifies a cloud service provider identifier (“CSP_ID”) to which thecloud exchange is to attach for service delivery, and a service key.Each endpoint description for the first and second endpoints alsoincludes endpoint specific data, such as a metro location identifier,port identifiers, data center identifiers, and so forth.

The policies in the Policy container may identify the configurationssettings that needs to be applied to the service being instantiated,examples of policies for said services including firewall rules, NATrules, encryption policies, WAN optimization policies, and so forth. Inother words, the policies determine how the services applied will beconfigured for application to the service traffic between the first andsecond endpoints for the micro-service. The QoS parameters in the QOScontainer specify the QoS to be applied to service traffic for themicro-service. The time range in the TimeRange container specifies astart time and end time that define a duration of the micro-service,during which the programmable network platform may provide assurance tothe originator and/or owner using the service identifier (“Srvc_ID”).

The interface for communicating service definitions according to thedata model may include eXtensible Markup Language (XML) or JavaScriptObject Notation (JSON) over HTTP/HTTPS. That is, the cloud exchangeprovider may define an XML/JSON interface that receives servicedefinitions according to the above-described exampled service data mode,and expose HTTP endpoints by which to receive such service definitions.

FIG. 15 is a conceptual diagram illustrating interfaces among componentsfor programming a cloud exchange using a programmable network platformaccording to techniques described in this disclosure. Programmablenetwork platform 12500 exposes a service API 12710 for service deliveryand data access. This various embodiments of APIs and other interfacesdescribed elsewhere in this disclosure for communicating withembodiments of programmable network platform 12500 may all representexamples of service API 12710. Service API 12710 may use a programmablenetwork service data model, such as MS_API described above, for definingan end-to-end service made up of a topology of micro-services.

Meta console 12704 represents a platform manufactured by the cloudservice provider, usable by cloud service provider technicians oroperators, e.g., that invokes the service API 12710 of programmablenetwork platform 12500. “CX” or customer portal 12702 represents aplatform manufactured by the cloud service provider, usable byenterprise/customer/CSP technicians or operators, e.g., that invokes theservice API 12710 of programmable network platform 12500. Cloud exchangedeveloper (CX API) 12700 represents third-party developed orcloud-service provider-developed platforms created by third-partydevelopers (e.g., CSP or customer developers) or cloud exchange providerdevelopers that invoke service API 12710 to request services from theprogrammable network platform 12500.

Business Applications 12706 may store accounting information forcustomers. For instance, Business Applications 12706 may store billinginformation for customers, such as name, customer identifier, address,phone number, email, to name only a few examples. When programmablenetwork platform 12500 configures a service for a customer that includesa service charge, Business Applications 12706 may store such expenseinformation. In this way, Business Applications 12706 may provide anaccounting of services purchased by a customer and provide unifiedbilling for such services. The API between Business Applications 12706and programmable network platform 12500 may include PACE integration.

FIG. 16 is a block diagram illustrating a programmable network platformthat includes interfaces by which external applications may configure acloud exchange to facilitate delivery of cloud services from cloudservice providers according to techniques described in this disclosure.In this example, programmable network platform 12500 exposes a serviceAPI 12820 for service delivery and data access. This various embodimentsof APIs and other interfaces described elsewhere in this disclosure forcommunicating with embodiments of programmable network platform 12500may all represent examples of service API 12820.

Service API 12820 includes, in this example, at least one third-partyplugin 12810 developed by cloud service providers and executed by theprogrammable network platform 12500 to request and establish layer 3cloud services from the cloud service providers. Plugin 12810 mayrepresent any of third-party orchestration modules 10404. The plugin12810 may implement a common plugin interface for the programmablenetwork platform 12500 and translate interface methods, fields, etc., toa cloud service provider interface for CSP orchestration. For example,programmable network platform 12500 may invoke plugin 12810 to request aservice instance from a cloud service provider for the cloud exchangeprovider (e.g., a 60 GB data storage service). Plugin 12810 for thecloud service provider receives the request and invokes CSPorchestration system 12800 to allow the cloud service provider toorchestrate the instantiation of the requested service. CSPorchestration 12800 via plugin 12810 then returns connectivityinformation in the form of a “network handle” to the programmablenetwork platform 12500. The network handle includes information by whichthe cloud exchange can connect to the instantiated, requested service.For example, the network handle may include a VxLAN or VLAN identifier,a layer 3 route or network address, tunnel information and/or cloudaggregate link information. The programmable network platform 12500 usesthe network handle to configure edge network 12600 to connect to theinstantiated, requested service, and to interconnect at least onecustomer network to the instantiated, requested service.

Operations portal 12804 represents a platform manufactured by the cloudservice provider, for use by cloud service provider 12804 technicians oroperators, e.g., that invokes the service API 12820 of programmablenetwork platform 12500. CSP orchestration system 12800 represents one ormore systems developed by the cloud service providers and usable by theprogrammable network platform 12500 to request layer 3 services from thecloud service providers. API gateway 12802 offers a high-level API bywhich customer-developed platforms or a cloud service provider-developedcustomer portal may request services from the programmable networkplatform 12500. Additional details of the API gateway and high-level APIare found in U.S. Provisional Patent Appln. No. 62/072,976, incorporatedabove.

FIG. 17 is a block diagram illustrating further details of one exampleof a computing device that operates in accordance with one or moretechniques of the present disclosure. FIG. 17 may illustrate aparticular example of a server or other computing device 13500 thatincludes one or more processor(s) 13502 for executing any one or more ofthe programmable network platform components (e.g., CNC, NFU, etc.), orany other system, application, or module described herein. Otherexamples of computing device 13500 may be used in other instances.Although shown in FIG. 17 as a stand-alone computing device 13500 forpurposes of example, a computing device may be any component or systemthat includes one or more processors or other suitable computingenvironment for executing software instructions and, for example, neednot necessarily include one or more elements shown in FIG. 17 (e.g.,communication units 13506; and in some examples components such asstorage device(s) 13508 may not be co-located or in the same chassis asother components).

As shown in the specific example of FIG. 17, computing device 13500includes one or more processors 13502, one or more input devices 13504,one or more communication units 13506, one or more output devices 13512,one or more storage devices 13508, and user interface (UI) device 13510,and communication unit 13506. Computing device 13500, in one example,further includes one or more applications 13522, programmable networkplatform application(s) 13524, and operating system 13516 that areexecutable by computing device 13500. Each of components 13502, 13504,13506, 13508, 13510, and 13512 are coupled (physically, communicatively,and/or operatively) for inter-component communications. In someexamples, communication channels 13514 may include a system bus, anetwork connection, an inter-process communication data structure, orany other method for communicating data. As one example, components13502, 13504, 13506, 13508, 13510, and 13512 may be coupled by one ormore communication channels 13514.

Processors 13502, in one example, are configured to implementfunctionality and/or process instructions for execution within computingdevice 13500. For example, processors 13502 may be capable of processinginstructions stored in storage device 13508. Examples of processors13502 may include, any one or more of a microprocessor, a controller, adigital signal processor (DSP), an application specific integratedcircuit (ASIC), a field-programmable gate array (FPGA), or equivalentdiscrete or integrated logic circuitry.

One or more storage devices 13508 may be configured to store informationwithin computing device 13500 during operation. Storage device 13508, insome examples, is described as a computer-readable storage medium. Insome examples, storage device 13508 is a temporary memory, meaning thata primary purpose of storage device 13508 is not long-term storage.Storage device 13508, in some examples, is described as a volatilememory, meaning that storage device 13508 does not maintain storedcontents when the computer is turned off. Examples of volatile memoriesinclude random access memories (RAM), dynamic random access memories(DRAM), static random access memories (SRAM), and other forms ofvolatile memories known in the art. In some examples, storage device13508 is used to store program instructions for execution by processors13502. Storage device 13508, in one example, is used by software orapplications running on computing device 13500 to temporarily storeinformation during program execution.

Storage devices 13508, in some examples, also include one or morecomputer-readable storage media. Storage devices 13508 may be configuredto store larger amounts of information than volatile memory. Storagedevices 13508 may further be configured for long-term storage ofinformation. In some examples, storage devices 13508 includenon-volatile storage elements. Examples of such non-volatile storageelements include magnetic hard discs, optical discs, floppy discs, flashmemories, or forms of electrically programmable memories (EPROM) orelectrically erasable and programmable (EEPROM) memories.

Computing device 13500, in some examples, also includes one or morecommunication units 13506. Computing device 13500, in one example,utilizes communication units 13506 to communicate with external devicesvia one or more networks, such as one or more wired/wireless/mobilenetworks. Communication units 13506 may include a network interfacecard, such as an Ethernet card, an optical transceiver, a radiofrequency transceiver, or any other type of device that can send andreceive information. In some examples, computing device 13500 usescommunication unit 13506 to communicate with an external device.

Computing device 13500, in one example, also includes one or more userinterface devices 13510. User interface devices 13510, in some examples,are configured to receive input from a user through tactile, audio, orvideo feedback. Examples of user interface devices(s) 13510 include apresence-sensitive display, a mouse, a keyboard, a voice responsivesystem, video camera, microphone or any other type of device fordetecting a command from a user. In some examples, a presence-sensitivedisplay includes a touch-sensitive screen.

One or more output devices 13512 may also be included in computingdevice 13500. Output device 13512, in some examples, is configured toprovide output to a user using tactile, audio, or video stimuli. Outputdevice 13512, in one example, includes a presence-sensitive display, asound card, a video graphics adapter card, or any other type of devicefor converting a signal into an appropriate form understandable tohumans or machines. Additional examples of output device 13512 include aspeaker, a cathode ray tube (CRT) monitor, a liquid crystal display(LCD), or any other type of device that can generate intelligible outputto a user.

Computing device 13500 may include operating system 13516. Operatingsystem 13516, in some examples, controls the operation of components ofcomputing device 13500. For example, operating system 13516, in oneexample, facilitates the communication of one or more applications 13522and programmable network platform application(s) 13524 with processors13502, communication unit 13506, storage device 13508, input device13504, user interface devices 13510, and output device 13512.

Application 522 and programmable network platform application(s) 13524may also include program instructions and/or data that are executable bycomputing device 13500. Example programmable network platformapplication(s) 13524 executable by computing device 13500 may includeany one or more of centralized network control application 13550 (“CNC13550”) and network field unit application 13552 (“NFU 13552”), eachillustrated with dashed lines to indicate that these may or may not beexecutable by any given example of computing device 13500.

Centralized network control 13550 may include instructions for causingcomputing device 13500 to perform one or more of the operations andactions described in the present disclosure with respect to centralizednetwork control. As one example, CNC 13550 may include instructions thatcause computing device 13500 to establish, de-install and manageinterconnections with multiple, different cloud service providersparticipating in the cloud exchange in an automated and seamless manner.

Network Field Unit 13552 may include instructions for causing computingdevice 13500 to perform one or more of the operations and actionsdescribed in the present disclosure with respect to network field units.As one example, NFU 13552 may include instructions that cause computingdevice 13500 to receive requests or instructions from a CNC executing onanother server, e.g., in a different geographically location such as adifferent data center, to configure network infrastructure of a cloudexchange point in order to provision one or more services.

The techniques described herein may be implemented in hardware,software, firmware, or any combination thereof. Various featuresdescribed as modules, units or components may be implemented together inan integrated logic device or separately as discrete but interoperablelogic devices or other hardware devices. In some cases, various featuresof electronic circuitry may be implemented as one or more integratedcircuit devices, such as an integrated circuit chip or chipset.

If implemented in hardware, this disclosure may be directed to anapparatus such as a processor or an integrated circuit device, such asan integrated circuit chip or chipset. Alternatively or additionally, ifimplemented in software or firmware, the techniques may be realized atleast in part by a computer-readable data storage medium comprisinginstructions that, when executed, cause a processor to perform one ormore of the methods described above. For example, the computer-readabledata storage medium may store such instructions for execution by aprocessor.

A computer-readable medium may form part of a computer program product,which may include packaging materials. A computer-readable medium maycomprise a computer data storage medium such as random access memory(RAM), read-only memory (ROM), non-volatile random access memory(NVRAM), electrically erasable programmable read-only memory (EEPROM),Flash memory, magnetic or optical data storage media, and the like. Insome examples, an article of manufacture may comprise one or morecomputer-readable storage media.

In some examples, the computer-readable storage media may comprisenon-transitory media. The term “non-transitory” may indicate that thestorage medium is not embodied in a carrier wave or a propagated signal.In certain examples, a non-transitory storage medium may store data thatcan, over time, change (e.g., in RAM or cache).

The code or instructions may be software and/or firmware executed byprocessing circuitry including one or more processors, such as one ormore digital signal processors (DSPs), general purpose microprocessors,application-specific integrated circuits (ASICs), field-programmablegate arrays (FPGAs), or other equivalent integrated or discrete logiccircuitry. Accordingly, the term “processor,” as used herein may referto any of the foregoing structure or any other structure suitable forimplementation of the techniques described herein. In addition, in someaspects, functionality described in this disclosure may be providedwithin software modules or hardware modules.

Various embodiments have been described. These and other embodiments arewithin the scope of the following examples.

What is claimed is:
 1. A method comprising: providing, by a centralizednetwork control (CNC) system, a software interface to receive servicerequests for configuration of services within an edge network of one ormore network data centers that are controlled by the CNC system;receiving, by the CNC system and via the software interface, a servicerequest to configure a service within the edge network of the networkdata center, wherein the edge network within the one or more networkdata centers connects through one or more switching fabrics of the oneor more network data centers; generating, by the CNC system and based onthe service request, a network service definition that specifies one ormore service requirements to implement the service; determining, by theCNC system and based on the network service definition, at least onenetwork field unit that is capable of servicing the service request,wherein the network field unit controls a portion of the edge network,wherein the network service definition is usable by the at least onenetwork field unit to configure the portion of the edge network toprovide the service; and sending, by the CNC system and to the at leastone network field unit, the network service definition to configure atleast one network service of the service at the portion of the edgenetwork.
 2. The method of claim 1, further comprising: monitoring, bythe CNC system, performance data for the service that is received fromthe at least one network field unit; comparing, by the CNC system, theperformance data with at least one of the service requirements andservice telemetry and analytics data stored by the CNC system todetermine whether the service requirements are satisfied by performanceof the portion of the edge network; responsive to determining that theservice requirements for the service are not satisfied, determining, bythe CNC system, at least one remedial action to satisfy the servicerequirements for the service; and executing, by the CNC system, the atleast one remedial action.
 3. The method of claim 2, wherein executingthe at least one remedial action comprises sending one or moreinstructions to the at least one network field unit to reconfigure atleast the portion of the edge network or a different portion of the edgenetwork.
 4. The method of any of claims 1, wherein the one or moreservice requirements specify at least one of a type of service, ageographic location for the service, a bandwidth for the service, anindication whether the bandwidth is burstable, an Excess InformationRate (EIR), a maximum latency for the bandwidth, a minimum availabilitylevel of the service, a cloud service provider for the service, or oneor more security properties for the service.
 5. The method of any ofclaims 1, wherein the service comprises at least one of a layer-3interconnection service, a firewall service, a data storage service, asoftware-as-a-service (SaaS) service, an analytics service, a networkaddress translation service, or a deep-packet inspection service.
 6. Themethod of any of claims 1, further comprising: responsive to configuringthe at least one network service of service at the portion of the edgenetwork, storing, by the CNC system, billing information that indicatesa fee for provisioning the service to a requestor of the service; andsending, by the CNC system, an indication of the billing information tothe requestor of the service.
 7. The method of any of claims 1, whereinthe one or more network data centers are geographically dispersed, andwherein a plurality of network field units are geographically dispersed,respectively, with the one or more geographically dispersed network datacenters, each of the plurality of network field units controlling arespective portion of the edge network.
 8. The method of any of claims1, wherein the service requirements of the service definition do notspecify particular, physical devices of the edge network, and whereinthe network service definition is usable by the at least one networkfield unit to configure the particular, physical devices of the edgenetwork.
 9. The method of any of claims 1, wherein the service is afirst service, wherein sending the network service definition toconfigure the at least one network service of the first service at theportion of the edge network, further comprises: configuring the edgenetwork to aggregate first cloud service traffic originating from afirst cloud service provider and second cloud service trafficoriginating from a second cloud service provider; and configuring theedge network to deliver the aggregated cloud service traffic.
 10. Acentralized network control (CNC) system comprising: one or morecomputer processors; and a memory comprising instructions that whenexecuted by the one or more computer processors cause the one or morecomputer processors to: provide a software interface to receive servicerequests for configuration of services within an edge network of one ormore network data centers that are controlled by the CNC system;receive, via the software interface, a service request to configure aservice within the edge network of the network data center, wherein theedge network within the one or more network data centers connectsthrough one or more switching fabrics of the one or more network datacenters; generate, based on the service request, a network servicedefinition that specifies one or more service requirements to implementthe service; determine, based on the network service definition, atleast one network field unit that is capable of servicing the servicerequest, wherein the network field unit controls a portion of the edgenetwork, wherein the network service definition is usable by the atleast one network field unit to configure the portion of the edgenetwork to provide the service; and send, to the at least one networkfield unit, the network service definition to configure at least onenetwork service of the service at the portion of the edge network. 11.The CNC system of claim 10, wherein the memory comprises instructionsthat when executed by the one or more computer processors cause the oneor more computer processors to: monitor performance data for the servicethat is received from the at least one network field unit; compare theperformance data with at least one of the service requirements andservice telemetry and analytics data stored by the CNC system todetermine whether the service requirements are satisfied by performanceof the portion of the edge network; responsive to determining that theservice requirements for the service are not satisfied, determine atleast one remedial action to satisfy the service requirements for theservice; and execute the at least one remedial action.
 12. The CNCsystem of claim 11, wherein executing the at least one remedial actioncomprises sending one or more instructions to the at least one networkfield unit to reconfigure at least the portion of the edge network or adifferent portion of the edge network.
 13. The CNC system of claim 10,wherein the one or more service requirements specify at least one of atype of service, a geographic location for the service, a bandwidth forthe service, an indication whether the bandwidth is burstable, an ExcessInformation Rate (EIR), a maximum latency for the bandwidth, a minimumavailability level of the service, a cloud service provider for theservice, or one or more security properties for the service.
 14. The CNCsystem of claim 10, wherein the service comprises at least one of alayer-3 interconnection service, a firewall service, a data storageservice, a software-as-a-service (SaaS) service, an analytics service, anetwork address translation service, or a deep-packet inspectionservice.
 15. The CNC system of claim 10, wherein the memory comprisesinstructions that when executed by the one or more computer processorscause the one or more computer processors to: responsive to configuringthe at least one network service of service at the portion of the edgenetwork, store billing information that indicates a fee for provisioningthe service to a requestor of the service; and send an indication of thebilling information to the requestor of the service.
 16. The CNC systemof claim 10, wherein the one or more network data centers aregeographically dispersed, and wherein a plurality of network field unitsare geographically dispersed, respectively, with the one or moregeographically dispersed network data centers, each of the plurality ofnetwork field units controlling a respective portion of the edgenetwork.
 17. The CNC system of claim 10, wherein the servicerequirements of the service definition do not specify particular,physical devices of the edge network, and wherein the network servicedefinition is usable by the at least one network field unit to configurethe particular, physical devices of the edge network.
 18. The CNC systemof claim 10, wherein the service is a first service, wherein the memorycomprises instructions that when executed by the one or more computerprocessors cause the one or more computer processors to: configure theedge network to aggregate first cloud service traffic originating from afirst cloud service provider and second cloud service trafficoriginating from a second cloud service provider; and configure the edgenetwork to deliver the aggregated cloud service traffic.
 19. Anon-transitory computer readable medium comprising instructions that,when executed, cause at least one programmable processor of acentralized network control (CNC) system to perform operationscomprising: providing a software interface to receive service requestsfor configuration of services within an edge network of one or morenetwork data centers that are controlled by the CNC system; receiving,via the software interface, a service request to configure a servicewithin the edge network of the network data center, wherein the edgenetwork within the one or more network data centers connects through oneor more switching fabrics of the one or more network data centers;generating, based on the service request, a network service definitionthat specifies one or more service requirements to implement theservice; determining, based on the network service definition, at leastone network field unit that is capable of servicing the service request,wherein the network field unit controls a portion of the edge network,wherein the network service definition is usable by the at least onenetwork field unit to configure the portion of the edge network toprovide the service; and sending, to the at least one network fieldunit, the network service definition to configure at least one networkservice of the service at the portion of the edge network.
 20. Thenon-transitory computer readable medium 19 comprising instructions that,when executed, cause the at least one programmable processor to performoperations comprising: monitoring performance data for the service thatis received from the at least one network field unit; comparing theperformance data with at least one of the service requirements andservice telemetry and analytics data stored by the CNC system todetermine whether the service requirements are satisfied by performanceof the portion of the edge network; responsive to determining that theservice requirements for the service are not satisfied, determining atleast one remedial action to satisfy the service requirements for theservice; and executing the at least one remedial action.